Problem Statement¶
Context¶
Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.
Objective¶
You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.
Data Dictionary¶
CustomerId: Unique ID which is assigned to each customer
Surname: Last name of the customer
CreditScore: It defines the credit history of the customer.
Geography: A customer’s location
Gender: It defines the Gender of the customer
Age: Age of the customer
Tenure: Number of years for which the customer has been with the bank
NumOfProducts: refers to the number of products that a customer has purchased through the bank.
Balance: Account balance
HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.
EstimatedSalary: Estimated salary
isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )
Exited : whether or not the customer left the bank within six month. It can take two values ** 0=No ( Customer did not leave the bank ) ** 1=Yes ( Customer left the bank )
Importing necessary libraries¶
!pip install tensorflow==2.18.0 scikit-learn==1.6.0 seaborn==0.13.1 matplotlib==3.8.1 numpy==1.26.2 pandas==2.2.2 imbalanced-learn==0.13.0 -q --user
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 1.8 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.5/13.5 MB 110.4 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 294.8/294.8 kB 22.6 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 94.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 93.2 MB/s eta 0:00:00 WARNING: The script f2py is installed in '/root/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
import pandas as pd # Library for data manipulation and analysis.
import numpy as np # Fundamental package for scientific computing.
import matplotlib.pyplot as plt # Plotting library for creating visualizations.
import seaborn as sns #For advanced visualizations.
from sklearn.model_selection import StratifiedKFold
from sklearn.utils.class_weight import compute_class_weight
from sklearn.model_selection import train_test_split # Function for splitting datasets for training and testing.
from sklearn.preprocessing import StandardScaler
import category_encoders as ce # Assuming you've installed category_encoders
import time # Module for time-related operations.
from imblearn.over_sampling import SMOTE # Import SMOTE from the correct module
from imblearn.under_sampling import RandomUnderSampler # Import RandomUnderSampler from the correct module
import tensorflow as tf #An end-to-end open source machine learning platform
tf.config.run_functions_eagerly(True)
from tensorflow import keras # High-level neural networks API for deep learning.
from keras import backend as K
from keras import backend # Abstraction layer for neural network backend engines.
from keras.models import Sequential # Model for building NN sequentially.
from keras.layers import Dense,Dropout,BatchNormalization # for creating fully connected neural network layers.
from keras import regularizers # Import regularizers
from tensorflow.keras.layers import LeakyReLU
from tensorflow.keras.layers import Input # Add this line to import Input
from tensorflow.keras.models import Sequential, Model # Update this line to include Model
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization # Update this line to include BatchNormalization
from keras.callbacks import EarlyStopping
!pip install category-encoders==2.6.2 --user
Collecting category-encoders==2.6.2 Downloading category_encoders-2.6.2-py2.py3-none-any.whl.metadata (8.0 kB) Requirement already satisfied: numpy>=1.14.0 in /root/.local/lib/python3.11/site-packages (from category-encoders==2.6.2) (1.26.2) Requirement already satisfied: scikit-learn>=0.20.0 in /root/.local/lib/python3.11/site-packages (from category-encoders==2.6.2) (1.6.0) Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from category-encoders==2.6.2) (1.13.1) Requirement already satisfied: statsmodels>=0.9.0 in /usr/local/lib/python3.11/dist-packages (from category-encoders==2.6.2) (0.14.4) Requirement already satisfied: pandas>=1.0.5 in /usr/local/lib/python3.11/dist-packages (from category-encoders==2.6.2) (2.2.2) Requirement already satisfied: patsy>=0.5.1 in /usr/local/lib/python3.11/dist-packages (from category-encoders==2.6.2) (1.0.1) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.0.5->category-encoders==2.6.2) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.0.5->category-encoders==2.6.2) (2025.1) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.0.5->category-encoders==2.6.2) (2025.1) Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.11/dist-packages (from scikit-learn>=0.20.0->category-encoders==2.6.2) (1.4.2) Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.11/dist-packages (from scikit-learn>=0.20.0->category-encoders==2.6.2) (3.5.0) Requirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.11/dist-packages (from statsmodels>=0.9.0->category-encoders==2.6.2) (24.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.8.2->pandas>=1.0.5->category-encoders==2.6.2) (1.17.0) Downloading category_encoders-2.6.2-py2.py3-none-any.whl (81 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.8/81.8 kB 2.3 MB/s eta 0:00:00 Installing collected packages: category-encoders Successfully installed category-encoders-2.6.2
!pip install category-encoders -q --user
Loading the dataset¶
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
data = pd.read_csv("/content/drive/MyDrive/Neural Network Bank Project/bank-1.csv")
df=data.copy()
Data Overview¶
df.head(25)
| RowNumber | CustomerId | Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 15634602 | Hargrave | 619 | France | Female | 42 | 2 | 0.00 | 1 | 1 | 1 | 101348.88 | 1 |
| 1 | 2 | 15647311 | Hill | 608 | Spain | Female | 41 | 1 | 83807.86 | 1 | 0 | 1 | 112542.58 | 0 |
| 2 | 3 | 15619304 | Onio | 502 | France | Female | 42 | 8 | 159660.80 | 3 | 1 | 0 | 113931.57 | 1 |
| 3 | 4 | 15701354 | Boni | 699 | France | Female | 39 | 1 | 0.00 | 2 | 0 | 0 | 93826.63 | 0 |
| 4 | 5 | 15737888 | Mitchell | 850 | Spain | Female | 43 | 2 | 125510.82 | 1 | 1 | 1 | 79084.10 | 0 |
| 5 | 6 | 15574012 | Chu | 645 | Spain | Male | 44 | 8 | 113755.78 | 2 | 1 | 0 | 149756.71 | 1 |
| 6 | 7 | 15592531 | Bartlett | 822 | France | Male | 50 | 7 | 0.00 | 2 | 1 | 1 | 10062.80 | 0 |
| 7 | 8 | 15656148 | Obinna | 376 | Germany | Female | 29 | 4 | 115046.74 | 4 | 1 | 0 | 119346.88 | 1 |
| 8 | 9 | 15792365 | He | 501 | France | Male | 44 | 4 | 142051.07 | 2 | 0 | 1 | 74940.50 | 0 |
| 9 | 10 | 15592389 | H? | 684 | France | Male | 27 | 2 | 134603.88 | 1 | 1 | 1 | 71725.73 | 0 |
| 10 | 11 | 15767821 | Bearce | 528 | France | Male | 31 | 6 | 102016.72 | 2 | 0 | 0 | 80181.12 | 0 |
| 11 | 12 | 15737173 | Andrews | 497 | Spain | Male | 24 | 3 | 0.00 | 2 | 1 | 0 | 76390.01 | 0 |
| 12 | 13 | 15632264 | Kay | 476 | France | Female | 34 | 10 | 0.00 | 2 | 1 | 0 | 26260.98 | 0 |
| 13 | 14 | 15691483 | Chin | 549 | France | Female | 25 | 5 | 0.00 | 2 | 0 | 0 | 190857.79 | 0 |
| 14 | 15 | 15600882 | Scott | 635 | Spain | Female | 35 | 7 | 0.00 | 2 | 1 | 1 | 65951.65 | 0 |
| 15 | 16 | 15643966 | Goforth | 616 | Germany | Male | 45 | 3 | 143129.41 | 2 | 0 | 1 | 64327.26 | 0 |
| 16 | 17 | 15737452 | Romeo | 653 | Germany | Male | 58 | 1 | 132602.88 | 1 | 1 | 0 | 5097.67 | 1 |
| 17 | 18 | 15788218 | Henderson | 549 | Spain | Female | 24 | 9 | 0.00 | 2 | 1 | 1 | 14406.41 | 0 |
| 18 | 19 | 15661507 | Muldrow | 587 | Spain | Male | 45 | 6 | 0.00 | 1 | 0 | 0 | 158684.81 | 0 |
| 19 | 20 | 15568982 | Hao | 726 | France | Female | 24 | 6 | 0.00 | 2 | 1 | 1 | 54724.03 | 0 |
| 20 | 21 | 15577657 | McDonald | 732 | France | Male | 41 | 8 | 0.00 | 2 | 1 | 1 | 170886.17 | 0 |
| 21 | 22 | 15597945 | Dellucci | 636 | Spain | Female | 32 | 8 | 0.00 | 2 | 1 | 0 | 138555.46 | 0 |
| 22 | 23 | 15699309 | Gerasimov | 510 | Spain | Female | 38 | 4 | 0.00 | 1 | 1 | 0 | 118913.53 | 1 |
| 23 | 24 | 15725737 | Mosman | 669 | France | Male | 46 | 3 | 0.00 | 2 | 0 | 1 | 8487.75 | 0 |
| 24 | 25 | 15625047 | Yen | 846 | France | Female | 38 | 5 | 0.00 | 1 | 1 | 1 | 187616.16 | 0 |
df.tail()
| RowNumber | CustomerId | Surname | CreditScore | Geography | Gender | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 9995 | 9996 | 15606229 | Obijiaku | 771 | France | Male | 39 | 5 | 0.00 | 2 | 1 | 0 | 96270.64 | 0 |
| 9996 | 9997 | 15569892 | Johnstone | 516 | France | Male | 35 | 10 | 57369.61 | 1 | 1 | 1 | 101699.77 | 0 |
| 9997 | 9998 | 15584532 | Liu | 709 | France | Female | 36 | 7 | 0.00 | 1 | 0 | 1 | 42085.58 | 1 |
| 9998 | 9999 | 15682355 | Sabbatini | 772 | Germany | Male | 42 | 3 | 75075.31 | 2 | 1 | 0 | 92888.52 | 1 |
| 9999 | 10000 | 15628319 | Walker | 792 | France | Female | 28 | 4 | 130142.79 | 1 | 1 | 0 | 38190.78 | 0 |
df.shape
(10000, 14)
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10000 entries, 0 to 9999 Data columns (total 14 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 RowNumber 10000 non-null int64 1 CustomerId 10000 non-null int64 2 Surname 10000 non-null object 3 CreditScore 10000 non-null int64 4 Geography 10000 non-null object 5 Gender 10000 non-null object 6 Age 10000 non-null int64 7 Tenure 10000 non-null int64 8 Balance 10000 non-null float64 9 NumOfProducts 10000 non-null int64 10 HasCrCard 10000 non-null int64 11 IsActiveMember 10000 non-null int64 12 EstimatedSalary 10000 non-null float64 13 Exited 10000 non-null int64 dtypes: float64(2), int64(9), object(3) memory usage: 1.1+ MB
df.describe()
| RowNumber | CustomerId | CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 10000.00000 | 1.000000e+04 | 10000.000000 | 10000.000000 | 10000.000000 | 10000.000000 | 10000.000000 | 10000.00000 | 10000.000000 | 10000.000000 | 10000.000000 |
| mean | 5000.50000 | 1.569094e+07 | 650.528800 | 38.921800 | 5.012800 | 76485.889288 | 1.530200 | 0.70550 | 0.515100 | 100090.239881 | 0.203700 |
| std | 2886.89568 | 7.193619e+04 | 96.653299 | 10.487806 | 2.892174 | 62397.405202 | 0.581654 | 0.45584 | 0.499797 | 57510.492818 | 0.402769 |
| min | 1.00000 | 1.556570e+07 | 350.000000 | 18.000000 | 0.000000 | 0.000000 | 1.000000 | 0.00000 | 0.000000 | 11.580000 | 0.000000 |
| 25% | 2500.75000 | 1.562853e+07 | 584.000000 | 32.000000 | 3.000000 | 0.000000 | 1.000000 | 0.00000 | 0.000000 | 51002.110000 | 0.000000 |
| 50% | 5000.50000 | 1.569074e+07 | 652.000000 | 37.000000 | 5.000000 | 97198.540000 | 1.000000 | 1.00000 | 1.000000 | 100193.915000 | 0.000000 |
| 75% | 7500.25000 | 1.575323e+07 | 718.000000 | 44.000000 | 7.000000 | 127644.240000 | 2.000000 | 1.00000 | 1.000000 | 149388.247500 | 0.000000 |
| max | 10000.00000 | 1.581569e+07 | 850.000000 | 92.000000 | 10.000000 | 250898.090000 | 4.000000 | 1.00000 | 1.000000 | 199992.480000 | 1.000000 |
df.describe(include=["object"]).T
| count | unique | top | freq | |
|---|---|---|---|---|
| Surname | 10000 | 2932 | Smith | 32 |
| Geography | 10000 | 3 | France | 5014 |
| Gender | 10000 | 2 | Male | 5457 |
df.isnull().sum()
| 0 | |
|---|---|
| RowNumber | 0 |
| CustomerId | 0 |
| Surname | 0 |
| CreditScore | 0 |
| Geography | 0 |
| Gender | 0 |
| Age | 0 |
| Tenure | 0 |
| Balance | 0 |
| NumOfProducts | 0 |
| HasCrCard | 0 |
| IsActiveMember | 0 |
| EstimatedSalary | 0 |
| Exited | 0 |
df.drop("RowNumber", axis=1, inplace=True)
df.drop("CustomerId", axis=1, inplace=True)
df.drop("Surname", axis=1, inplace=True)
df.nunique()
| 0 | |
|---|---|
| CreditScore | 460 |
| Geography | 3 |
| Gender | 2 |
| Age | 70 |
| Tenure | 11 |
| Balance | 6382 |
| NumOfProducts | 4 |
| HasCrCard | 2 |
| IsActiveMember | 2 |
| EstimatedSalary | 9999 |
| Exited | 2 |
#run a for loop that looks for all the uniqe value of specified datatypes
sanity_check = df.select_dtypes(include=['int64','object']).columns
for var in sanity_check:
print(var)
print(df[var].unique())
print()
CreditScore [619 608 502 699 850 645 822 376 501 684 528 497 476 549 635 616 653 587 726 732 636 510 669 846 577 756 571 574 411 591 533 553 520 722 475 490 804 582 472 465 556 834 660 776 829 637 550 698 585 788 655 601 656 725 511 614 742 687 555 603 751 581 735 661 675 738 813 657 604 519 664 678 757 416 665 777 543 506 493 652 750 729 646 647 808 524 769 730 515 773 814 710 413 623 670 622 785 605 479 685 538 562 721 628 668 828 674 625 432 770 758 795 686 789 589 461 584 579 663 682 793 691 485 650 754 535 716 539 706 586 631 717 800 683 704 615 667 484 480 578 512 606 597 778 514 525 715 580 807 521 759 516 711 618 643 671 689 620 676 572 695 592 567 694 547 594 673 610 767 763 712 703 662 659 523 772 545 634 739 771 681 544 696 766 727 693 557 531 498 651 791 733 811 707 714 782 775 799 602 744 588 747 583 627 731 629 438 642 806 474 559 429 680 749 734 644 626 649 805 718 840 630 654 762 568 613 522 737 648 443 640 540 460 593 801 611 802 745 483 690 492 709 705 560 752 701 537 487 596 702 486 724 548 464 790 534 748 494 590 468 509 818 816 536 753 774 621 569 658 798 641 542 692 639 765 570 638 599 632 779 527 564 833 504 842 508 417 598 741 607 761 848 546 439 755 760 526 713 700 666 566 495 688 612 477 427 839 819 720 459 503 624 529 563 482 796 445 746 786 554 672 787 499 844 450 815 838 803 736 633 600 679 517 792 743 488 421 841 708 507 505 456 435 561 518 565 728 784 552 609 764 697 723 551 444 719 496 541 830 812 677 420 595 617 809 500 826 434 513 478 797 363 399 463 780 452 575 837 794 824 428 823 781 849 489 431 457 768 831 359 820 573 576 558 817 449 440 415 821 530 350 446 425 740 481 783 358 845 451 458 469 423 404 836 473 835 466 491 351 827 843 365 532 414 453 471 401 810 832 470 447 422 825 430 436 426 408 847 418 437 410 454 407 455 462 386 405 383 395 467 433 442 424 448 441 367 412 382 373 419] Geography ['France' 'Spain' 'Germany'] Gender ['Female' 'Male'] Age [42 41 39 43 44 50 29 27 31 24 34 25 35 45 58 32 38 46 36 33 40 51 61 49 37 19 66 56 26 21 55 75 22 30 28 65 48 52 57 73 47 54 72 20 67 79 62 53 80 59 68 23 60 70 63 64 18 82 69 74 71 76 77 88 85 84 78 81 92 83] Tenure [ 2 1 8 7 4 6 3 10 5 9 0] NumOfProducts [1 3 2 4] HasCrCard [1 0] IsActiveMember [1 0] Exited [1 0]
#for loop that gives a count of unique values for desired data type
for i in df.describe(include=["object"]).columns:
print("Unique values in", i, "are :")
print(df[i].value_counts())
print("*" * 50)
Unique values in Geography are : Geography France 5014 Germany 2509 Spain 2477 Name: count, dtype: int64 ************************************************** Unique values in Gender are : Gender Male 5457 Female 4543 Name: count, dtype: int64 **************************************************
#for loop that gives a count of unique values for desired data type
for i in df.describe(include=["int64"]).columns:
print("Unique values in", i, "are :")
print(df[i].value_counts())
print("*" * 50)
Unique values in CreditScore are :
CreditScore
850 233
678 63
655 54
705 53
667 53
...
404 1
351 1
365 1
417 1
419 1
Name: count, Length: 460, dtype: int64
**************************************************
Unique values in Age are :
Age
37 478
38 477
35 474
36 456
34 447
...
92 2
82 1
88 1
85 1
83 1
Name: count, Length: 70, dtype: int64
**************************************************
Unique values in Tenure are :
Tenure
2 1048
1 1035
7 1028
8 1025
5 1012
3 1009
4 989
9 984
6 967
10 490
0 413
Name: count, dtype: int64
**************************************************
Unique values in NumOfProducts are :
NumOfProducts
1 5084
2 4590
3 266
4 60
Name: count, dtype: int64
**************************************************
Unique values in HasCrCard are :
HasCrCard
1 7055
0 2945
Name: count, dtype: int64
**************************************************
Unique values in IsActiveMember are :
IsActiveMember
1 5151
0 4849
Name: count, dtype: int64
**************************************************
Unique values in Exited are :
Exited
0 7963
1 2037
Name: count, dtype: int64
**************************************************
for i in df.describe(include=["float64"]).columns:
print("Unique values in", i, "are :")
print(df[i].value_counts())
print("*" * 50)
Unique values in Balance are :
Balance
0.00 3617
130170.82 2
105473.74 2
85304.27 1
159397.75 1
...
81556.89 1
112687.69 1
108698.96 1
238387.56 1
130142.79 1
Name: count, Length: 6382, dtype: int64
**************************************************
Unique values in EstimatedSalary are :
EstimatedSalary
24924.92 2
101348.88 1
55313.44 1
72500.68 1
182692.80 1
..
120893.07 1
188377.21 1
55902.93 1
4523.74 1
38190.78 1
Name: count, Length: 9999, dtype: int64
**************************************************
Exploratory Data Analysis¶
Univariate Analysis¶
# function to plot a boxplot and a histogram along the same scale.
def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
"""
Boxplot and histogram combined
data: dataframe
feature: dataframe column
figsize: size of figure (default (12,7))
kde: whether to the show density curve (default False)
bins: number of bins for histogram (default None)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(
nrows=2, # Number of rows of the subplot grid= 2
sharex=True, # x-axis will be shared among all subplots
gridspec_kw={"height_ratios": (0.25, 0.75)},
figsize=figsize,
) # creating the 2 subplots
sns.boxplot(
data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
) # boxplot will be created and a triangle will indicate the mean value of the column
sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
) if bins else sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2
) # For histogram
ax_hist2.axvline(
data[feature].mean(), color="green", linestyle="--"
) # Add mean to the histogram
ax_hist2.axvline(
data[feature].median(), color="black", linestyle="-"
) # Add median to the histogram
# function to create labeled barplots
def labeled_barplot(data, feature, perc=False, n=None):
"""
Barplot with percentage at the top
data: dataframe
feature: dataframe column
perc: whether to display percentages instead of count (default is False)
n: displays the top n category levels (default is None, i.e., display all levels)
"""
total = len(data[feature]) # length of the column
count = data[feature].nunique()
if n is None:
plt.figure(figsize=(count + 1, 5))
else:
plt.figure(figsize=(n + 1, 5))
plt.xticks(rotation=90, fontsize=15)
ax = sns.countplot(
data=data,
x=feature,
palette="Paired",
order=data[feature].value_counts().index[:n].sort_values(),
)
for p in ax.patches:
if perc == True:
label = "{:.1f}%".format(
100 * p.get_height() / total
) # percentage of each class of the category
else:
label = p.get_height() # count of each level of the category
x = p.get_x() + p.get_width() / 2 # width of the plot
y = p.get_height() # height of the plot
ax.annotate(
label,
(x, y),
ha="center",
va="center",
size=12,
xytext=(0, 5),
textcoords="offset points",
) # annotate the percentage
plt.show() # show the plot
# function to plot stacked bar chart
def stacked_barplot(data, predictor, target):
"""
Print the category counts and plot a stacked bar chart
data: dataframe
predictor: independent variable
target: target variable
"""
count = data[predictor].nunique()
sorter = data[target].value_counts().index[-1]
tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
by=sorter, ascending=False
)
print(tab1)
print("-" * 120)
tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
by=sorter, ascending=False
)
tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
plt.legend(
loc="lower left", frameon=False,
)
plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
plt.show()
### Function to plot distributions
def distribution_plot_wrt_target(data, predictor, target):
fig, axs = plt.subplots(2, 2, figsize=(12, 10))
target_uniq = data[target].unique()
axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
sns.histplot(
data=data[data[target] == target_uniq[0]],
x=predictor,
kde=True,
ax=axs[0, 0],
color="teal",
)
axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
sns.histplot(
data=data[data[target] == target_uniq[1]],
x=predictor,
kde=True,
ax=axs[0, 1],
color="orange",
)
axs[1, 0].set_title("Boxplot w.r.t target")
sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
sns.boxplot(
data=data,
x=target,
y=predictor,
ax=axs[1, 1],
showfliers=False,
palette="gist_rainbow",
)
plt.tight_layout()
plt.show()
def f1_minority(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
recall = true_positives / (possible_positives + K.epsilon())
f1_val = 2 * ((precision * recall) / (precision + recall + K.epsilon()))
return f1_val
def f1_minority(y_true, y_pred):
y_true = tf.cast(y_true, tf.float32) # Cast y_true to float32
y_pred = tf.cast(y_pred, tf.float32) # Cast y_pred to float32
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
recall = true_positives / (possible_positives + K.epsilon())
f1_val = 2 * ((precision * recall) / (precision + recall + K.epsilon()))
return f1_val
def f1_minority(y_true, y_pred):
y_true = tf.cast(y_true, tf.float32)
y_pred = tf.cast(y_pred, tf.float32)
tp = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
pp = K.sum(K.round(K.clip(y_pred, 0, 1)))
fp = pp - tp
fn = K.sum(K.round(K.clip(y_true, 0, 1))) - tp
precision = tp / (tp + fp + K.epsilon())
recall = tp / (tp + fn + K.epsilon())
f1_val = 2 * ((precision * recall) / (precision + recall + K.epsilon()))
return f1_val
labeled_barplot(df, 'Exited')
labeled_barplot(df, 'Geography')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. ax = sns.countplot(
labeled_barplot(df, 'Gender')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. ax = sns.countplot(
labeled_barplot(df, 'NumOfProducts')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. ax = sns.countplot(
labeled_barplot(df, 'HasCrCard')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. ax = sns.countplot(
labeled_barplot(df, 'IsActiveMember')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. ax = sns.countplot(
labeled_barplot(df, 'Age')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. ax = sns.countplot(
histogram_boxplot(df,'Age', kde=True)
labeled_barplot(df, 'Tenure')
<ipython-input-181-1e58b03272cc>:22: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. ax = sns.countplot(
histogram_boxplot(df,'Tenure', kde=True)
histogram_boxplot(df,'CreditScore', kde=True)
histogram_boxplot(df,'Balance', kde=True)
histogram_boxplot(df,'EstimatedSalary', kde=True)
Bivariate Analysis¶
sns.pairplot(df, hue='Exited')
<seaborn.axisgrid.PairGrid at 0x7a5a7072ecd0>
plt.figure(figsize=(15, 7))
sns.heatmap(data.corr(numeric_only = True), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="coolwarm")
plt.show()
Two of our x's corolate with our y somehat already thats good
pd.crosstab(df["Age"], df["Exited"]).plot(kind="bar")
plt.show()
stacked_barplot(df,"Age","Exited")
Exited 0 1 All Age All 7963 2037 10000 46 135 91 226 40 343 89 432 43 209 88 297 45 142 87 229 .. ... ... ... 79 4 0 4 78 5 0 5 77 10 0 10 76 11 0 11 75 9 0 9 [71 rows x 3 columns] ------------------------------------------------------------------------------------------------------------------------
distribution_plot_wrt_target(df,"Age","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow") <ipython-input-183-3732c5f6b024>:31: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(
pd.crosstab(df["Gender"], df["Exited"]).plot(kind="bar")
plt.show()
stacked_barplot(df,"Gender","Exited")
Exited 0 1 All Gender All 7963 2037 10000 Female 3404 1139 4543 Male 4559 898 5457 ------------------------------------------------------------------------------------------------------------------------
pd.crosstab(df["Geography"], df["Exited"]).plot(kind="bar")
plt.show()
stacked_barplot(df,"Geography","Exited")
Exited 0 1 All Geography All 7963 2037 10000 Germany 1695 814 2509 France 4204 810 5014 Spain 2064 413 2477 ------------------------------------------------------------------------------------------------------------------------
pd.crosstab(df["NumOfProducts"], df["Exited"]).plot(kind="bar")
plt.show()
stacked_barplot(df,"NumOfProducts","Exited")
Exited 0 1 All NumOfProducts All 7963 2037 10000 1 3675 1409 5084 2 4242 348 4590 3 46 220 266 4 0 60 60 ------------------------------------------------------------------------------------------------------------------------
distribution_plot_wrt_target(df,"NumOfProducts","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow") <ipython-input-183-3732c5f6b024>:31: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(
pd.crosstab(df["HasCrCard"], df["Exited"]).plot(kind="bar")
plt.show()
stacked_barplot(df,"HasCrCard","Exited")
Exited 0 1 All HasCrCard All 7963 2037 10000 1 5631 1424 7055 0 2332 613 2945 ------------------------------------------------------------------------------------------------------------------------
pd.crosstab(df["IsActiveMember"], df["Exited"]).plot(kind="bar")
plt.show()
stacked_barplot(df,"IsActiveMember","Exited")
Exited 0 1 All IsActiveMember All 7963 2037 10000 0 3547 1302 4849 1 4416 735 5151 ------------------------------------------------------------------------------------------------------------------------
pd.crosstab(df["Tenure"], df["Exited"]).plot(kind="bar")
plt.show()
stacked_barplot(df,"Tenure","Exited")
Exited 0 1 All Tenure All 7963 2037 10000 1 803 232 1035 3 796 213 1009 9 771 213 984 5 803 209 1012 4 786 203 989 2 847 201 1048 8 828 197 1025 6 771 196 967 7 851 177 1028 10 389 101 490 0 318 95 413 ------------------------------------------------------------------------------------------------------------------------
distribution_plot_wrt_target(df,"Tenure","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow") <ipython-input-183-3732c5f6b024>:31: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(
distribution_plot_wrt_target(df,"CreditScore","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow") <ipython-input-183-3732c5f6b024>:31: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(
pd.crosstab(df["CreditScore"], df["Exited"]).plot(kind="bar")
plt.show()
stacked_barplot(df,"CreditScore","Exited")
Exited 0 1 All CreditScore All 7963 2037 10000 850 190 43 233 651 33 17 50 705 37 16 53 637 32 14 46 ... ... ... ... 810 6 0 6 442 1 0 1 488 10 0 10 814 11 0 11 472 8 0 8 [461 rows x 3 columns] ------------------------------------------------------------------------------------------------------------------------
distribution_plot_wrt_target(df,"Balance","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow") <ipython-input-183-3732c5f6b024>:31: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(
stacked_barplot(df,"Balance","Exited")
Exited 0 1 All Balance All 7963 2037 10000 0.0 3117 500 3617 12459.19 0 1 1 116973.26 0 1 1 116755.5 0 1 1 ... ... ... ... 117864.85 1 0 1 117852.26 1 0 1 85982.07 1 0 1 85996.19 1 0 1 127146.68 1 0 1 [6383 rows x 3 columns] ------------------------------------------------------------------------------------------------------------------------
distribution_plot_wrt_target(df,"EstimatedSalary","Exited")
<ipython-input-183-3732c5f6b024>:28: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow") <ipython-input-183-3732c5f6b024>:31: FutureWarning: Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect. sns.boxplot(
stacked_barplot(df,"EstimatedSalary","Exited")
Exited 0 1 All EstimatedSalary All 7963 2037 10000 93132.61 0 1 1 91167.19 0 1 1 91556.57 0 1 1 91560.63 0 1 1 ... ... ... ... 23237.25 1 0 1 90384.26 1 0 1 90346.1 1 0 1 90305.97 1 0 1 100236.02 1 0 1 [10000 rows x 3 columns] ------------------------------------------------------------------------------------------------------------------------
Data Preprocessing¶
Dummy variables to prevent leakage
df = pd.get_dummies(df,columns=df.select_dtypes(include=["object"]).columns.tolist(),drop_first=True)
df = df.astype(float)
df.head()
| CreditScore | Age | Tenure | Balance | NumOfProducts | HasCrCard | IsActiveMember | EstimatedSalary | Exited | Geography_Germany | Geography_Spain | Gender_Male | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 619.0 | 42.0 | 2.0 | 0.00 | 1.0 | 1.0 | 1.0 | 101348.88 | 1.0 | 0.0 | 0.0 | 0.0 |
| 1 | 608.0 | 41.0 | 1.0 | 83807.86 | 1.0 | 0.0 | 1.0 | 112542.58 | 0.0 | 0.0 | 1.0 | 0.0 |
| 2 | 502.0 | 42.0 | 8.0 | 159660.80 | 3.0 | 1.0 | 0.0 | 113931.57 | 1.0 | 0.0 | 0.0 | 0.0 |
| 3 | 699.0 | 39.0 | 1.0 | 0.00 | 2.0 | 0.0 | 0.0 | 93826.63 | 0.0 | 0.0 | 0.0 | 0.0 |
| 4 | 850.0 | 43.0 | 2.0 | 125510.82 | 1.0 | 1.0 | 1.0 | 79084.10 | 0.0 | 0.0 | 1.0 | 0.0 |
Train-validation-test Split¶
## Storing required categorical variables to apply dummification
cols_list = ['CreditScore', 'Age', 'Tenure', 'Balance', 'NumOfProducts', 'EstimatedSalary']
X = df.drop(['Exited'],axis=1)
y = df['Exited']
print(X.shape)
print(y.shape)
print(type(X))
(10000, 11) (10000,) <class 'pandas.core.frame.DataFrame'>
# Splitting the dataset into the Training and Testing set.
X_large, X_test, y_large, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1,stratify=y,shuffle = True)
X_train, X_val, y_train, y_val = train_test_split(X_large, y_large, test_size = 0.2, random_state = 1,stratify=y_large, shuffle = True)
print(X_train.shape, X_val.shape, X_test.shape)
(6400, 11) (1600, 11) (2000, 11)
print(y_train.shape, y_val.shape, y_test.shape)
(6400,) (1600,) (2000,)
Data Normalization¶
# creating an instance of the standard scaler
sc = StandardScaler()
X_train[cols_list] = sc.fit_transform(X_train[cols_list])
X_val[cols_list] = sc.transform(X_val[cols_list])
X_test[cols_list] = sc.transform(X_test[cols_list])
i will only do batch normalization on some models and layers but not yet
Model Building¶
Model Evaluation Criterion¶
# defining a function to compute different metrics to check performance of a classification model built using sklearn
def model_performance_classification_sklearn(model, predictors, target):
"""
Function to compute different metrics to check classification model performance
model: classifier
predictors: independent variables
target: dependent variable
"""
# predicting using the independent variables
pred = model.predict(predictors)
acc = accuracy_score(target, pred) # to compute Accuracy
recall = recall_score(target, pred) # to compute Recall
precision = precision_score(target, pred) # to compute Precision
f1 = f1_score(target, pred) # to compute F1-score
# creating a dataframe of metrics
df_perf = pd.DataFrame(
{
"Accuracy": acc,
"Recall": recall,
"Precision": precision,
"F1": f1
},
index=[0],
)
return df_perf
def plot(history, name):
"""
Function to plot loss/accuracy
history: an object which stores the metrics and losses.
name: can be one of Loss or Accuracy
"""
fig, ax = plt.subplots() #Creating a subplot with figure and axes.
plt.plot(history.history[name]) #Plotting the train accuracy or train loss
plt.plot(history.history['val_'+name]) #Plotting the validation accuracy or validation loss
plt.title('Model ' + name.capitalize()) #Defining the title of the plot.
plt.ylabel(name.capitalize()) #Capitalizing the first letter.
plt.xlabel('Epoch') #Defining the label for the x-axis.
fig.legend(['Train', 'Validation'], loc="outside right upper") #Defining the legend, loc controls the position of the legend.
# Defining the columns of the dataframe which are nothing but the hyper parameters and the metrics.
columns = ["# hidden layers", "# neurons - hidden layer", "activation function - hidden layer", "# epochs", "batch size", "optimizer", "learning rate", "initializer", "regularizer", "train loss", "validation loss", "train recall", "validation recall", "time (secs)"]
# Creating a pandas dataframe.
results = pd.DataFrame(columns=columns)
Write down the logic for choosing the metric that would be the best metric for this business scenario.
Recall is the preferred metric because we want to minimize false negatives, ensuring that as many potential churners as possible are identified. we want to make personalized offers or some kind of outreach. The bottom line wont hurt too much if we send out a few extra offers.
Neural Network with SGD Optimizer¶
tf.keras.backend.clear_session()
model_0 = Sequential()
model_0.add(Dense(128, activation='relu', input_dim=X_train.shape[1]))
model_0.add(Dense(64, activation='relu'))
model_0.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001)
model_0.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
start = time.time()
history = model_0.fit(X_train, y_train, validation_data=(X_val, y_val), batch_size=16, epochs=10)
end = time.time()
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Epoch 1/10
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
399/400 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step - loss: 0.6929 - recall: 0.6014
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 16s 36ms/step - loss: 0.6927 - recall: 0.6001 - val_loss: 0.5907 - val_recall: 0.0031 Epoch 2/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 36ms/step - loss: 0.5748 - recall: 0.0029 - val_loss: 0.5326 - val_recall: 0.0000e+00 Epoch 3/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 36ms/step - loss: 0.5229 - recall: 0.0000e+00 - val_loss: 0.5067 - val_recall: 0.0000e+00 Epoch 4/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.5002 - recall: 0.0000e+00 - val_loss: 0.4926 - val_recall: 0.0000e+00 Epoch 5/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 36ms/step - loss: 0.4874 - recall: 0.0000e+00 - val_loss: 0.4834 - val_recall: 0.0000e+00 Epoch 6/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.4857 - recall: 0.0000e+00 - val_loss: 0.4763 - val_recall: 0.0000e+00 Epoch 7/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.4822 - recall: 0.0000e+00 - val_loss: 0.4703 - val_recall: 0.0000e+00 Epoch 8/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 36ms/step - loss: 0.4767 - recall: 0.0000e+00 - val_loss: 0.4650 - val_recall: 0.0000e+00 Epoch 9/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.4605 - recall: 0.0000e+00 - val_loss: 0.4602 - val_recall: 0.0000e+00 Epoch 10/10 400/400 ━━━━━━━━━━━━━━━━━━━━ 14s 35ms/step - loss: 0.4683 - recall: 0.0000e+00 - val_loss: 0.4558 - val_recall: 0.0031
print("Time taken in seconds ",end-start)
Time taken in seconds 143.8668487071991
plot(history,'loss')
it looks like 10 epochs is not enough its possoible the momentum is pushing it out of helpful minima
plot(history,'recall')
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt
# Generate predictions on the validation set using model_0
val_pred = model_0.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
38/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step precision recall f1-score support 0.0 0.80 1.00 0.89 1274 1.0 1.00 0.00 0.01 326 accuracy 0.80 1600 macro avg 0.90 0.50 0.45 1600 weighted avg 0.84 0.80 0.71 1600
here there is a major class imbalance which is severely impacting model performance when it comes to recall of our target variable. not sure whats going on with the heat map
results.loc[0] = [2, "128,64", "relu,relu", 10, 16, "sgd", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.0 | 0.003067 | 143.87 |
Model Performance Improvement¶
Neural Network with Adam Optimizer¶
tf.keras.backend.clear_session()
model_1 = Sequential()
# Get the correct input shape from your scaled training data
input_shape = X_train.shape[1]
# Input layer with 64 neurons and ReLU activation
model_1.add(Dense(64, activation='relu', input_shape=(input_shape,))) # Use input_shape
# Hidden layer with 32 neurons and ReLU activation
model_1.add(Dense(32, activation='relu'))
# Output layer with 1 neuron and sigmoid activation
model_1.add(Dense(1, activation='sigmoid'))
# Compile the model
optimizer = keras.optimizers.Adam(clipnorm=1.0)
model_1.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
# Train the model (adjust epochs and batch size as needed)
history = model_1.fit(X_train, y_train, validation_data=(X_val, y_val),
batch_size=16, epochs=50)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 0s 60ms/step - loss: 0.4753 - recall: 0.0955
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 27s 65ms/step - loss: 0.4751 - recall: 0.0957 - val_loss: 0.3744 - val_recall: 0.3313 Epoch 2/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 27s 67ms/step - loss: 0.3817 - recall: 0.3819 - val_loss: 0.3568 - val_recall: 0.4387 Epoch 3/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3547 - recall: 0.4532 - val_loss: 0.3584 - val_recall: 0.4693 Epoch 4/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.3382 - recall: 0.4808 - val_loss: 0.3518 - val_recall: 0.4663 Epoch 5/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.3319 - recall: 0.4711 - val_loss: 0.3500 - val_recall: 0.4755 Epoch 6/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3318 - recall: 0.5027 - val_loss: 0.3492 - val_recall: 0.4417 Epoch 7/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3240 - recall: 0.4521 - val_loss: 0.3487 - val_recall: 0.4264 Epoch 8/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3225 - recall: 0.4859 - val_loss: 0.3492 - val_recall: 0.5000 Epoch 9/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.3222 - recall: 0.5052 - val_loss: 0.3514 - val_recall: 0.4294 Epoch 10/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3255 - recall: 0.5173 - val_loss: 0.3481 - val_recall: 0.5245 Epoch 11/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.3149 - recall: 0.5226 - val_loss: 0.3500 - val_recall: 0.4417 Epoch 12/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3178 - recall: 0.4986 - val_loss: 0.3510 - val_recall: 0.4356 Epoch 13/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.2992 - recall: 0.5288 - val_loss: 0.3530 - val_recall: 0.4969 Epoch 14/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.3210 - recall: 0.4959 - val_loss: 0.3723 - val_recall: 0.3497 Epoch 15/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3249 - recall: 0.5259 - val_loss: 0.3643 - val_recall: 0.3834 Epoch 16/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3098 - recall: 0.5158 - val_loss: 0.3490 - val_recall: 0.4755 Epoch 17/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3114 - recall: 0.5318 - val_loss: 0.3550 - val_recall: 0.4080 Epoch 18/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3202 - recall: 0.5158 - val_loss: 0.3562 - val_recall: 0.4264 Epoch 19/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2973 - recall: 0.5487 - val_loss: 0.3615 - val_recall: 0.4233 Epoch 20/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.3043 - recall: 0.5353 - val_loss: 0.3534 - val_recall: 0.4724 Epoch 21/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2971 - recall: 0.5711 - val_loss: 0.3549 - val_recall: 0.4693 Epoch 22/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3036 - recall: 0.5502 - val_loss: 0.3570 - val_recall: 0.4479 Epoch 23/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.3027 - recall: 0.5286 - val_loss: 0.3562 - val_recall: 0.4939 Epoch 24/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2968 - recall: 0.5554 - val_loss: 0.3580 - val_recall: 0.4356 Epoch 25/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2885 - recall: 0.5471 - val_loss: 0.3584 - val_recall: 0.4601 Epoch 26/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2933 - recall: 0.5595 - val_loss: 0.3600 - val_recall: 0.4479 Epoch 27/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 27s 66ms/step - loss: 0.2843 - recall: 0.5835 - val_loss: 0.3571 - val_recall: 0.4632 Epoch 28/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2783 - recall: 0.5796 - val_loss: 0.3624 - val_recall: 0.4509 Epoch 29/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2821 - recall: 0.5689 - val_loss: 0.3608 - val_recall: 0.4417 Epoch 30/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2880 - recall: 0.5597 - val_loss: 0.3639 - val_recall: 0.4509 Epoch 31/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2758 - recall: 0.5944 - val_loss: 0.3724 - val_recall: 0.4724 Epoch 32/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2913 - recall: 0.5586 - val_loss: 0.3665 - val_recall: 0.5307 Epoch 33/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2723 - recall: 0.5827 - val_loss: 0.3621 - val_recall: 0.5460 Epoch 34/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.2826 - recall: 0.5878 - val_loss: 0.3630 - val_recall: 0.4724 Epoch 35/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2680 - recall: 0.5896 - val_loss: 0.3672 - val_recall: 0.4969 Epoch 36/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2780 - recall: 0.5743 - val_loss: 0.3727 - val_recall: 0.5031 Epoch 37/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 27s 66ms/step - loss: 0.2689 - recall: 0.5918 - val_loss: 0.3679 - val_recall: 0.4816 Epoch 38/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2730 - recall: 0.5957 - val_loss: 0.3770 - val_recall: 0.3957 Epoch 39/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2761 - recall: 0.5741 - val_loss: 0.3675 - val_recall: 0.4693 Epoch 40/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2698 - recall: 0.5916 - val_loss: 0.3697 - val_recall: 0.4693 Epoch 41/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2689 - recall: 0.6139 - val_loss: 0.3755 - val_recall: 0.4755 Epoch 42/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2538 - recall: 0.6080 - val_loss: 0.3721 - val_recall: 0.5337 Epoch 43/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 64ms/step - loss: 0.2725 - recall: 0.5899 - val_loss: 0.3834 - val_recall: 0.5460 Epoch 44/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2719 - recall: 0.6008 - val_loss: 0.3764 - val_recall: 0.4202 Epoch 45/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2619 - recall: 0.5966 - val_loss: 0.3861 - val_recall: 0.5092 Epoch 46/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2633 - recall: 0.6177 - val_loss: 0.3806 - val_recall: 0.4325 Epoch 47/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2642 - recall: 0.5845 - val_loss: 0.3748 - val_recall: 0.4693 Epoch 48/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 66ms/step - loss: 0.2705 - recall: 0.5773 - val_loss: 0.3801 - val_recall: 0.4663 Epoch 49/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2659 - recall: 0.6079 - val_loss: 0.3870 - val_recall: 0.4816 Epoch 50/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 26s 65ms/step - loss: 0.2452 - recall: 0.6384 - val_loss: 0.3858 - val_recall: 0.4571
print("Time taken in seconds ",end-start)
Time taken in seconds 606.9027745723724
plot(history,'loss')
better but more overfitting
plot(history,'recall')
more oscilations but better performance
val_pred = model_1.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
40/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step precision recall f1-score support 0.0 0.87 0.95 0.91 1274 1.0 0.70 0.46 0.55 326 accuracy 0.85 1600 macro avg 0.78 0.70 0.73 1600 weighted avg 0.84 0.85 0.84 1600
adam does better with the same batch size and number of epochs even with less complexity in term,s of parameters! at least on this dataset with class imbalances of this kind and of this size and complexity.
results.loc[1] = [2, "64,32", "relu,relu", 50, 16, "Adam", "0, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
we need to make sure we dont negelect precision at the expense of recall too much so we need to try both recall and accuracy and f1 score as our metric
tf.keras.backend.clear_session()
model_2 = Sequential()
model_2.add(Dense(128, activation="relu", input_dim=X_train.shape[1]))
model_2.add(BatchNormalization()) # Batch Normalization layer
model_2.add(Dense(64, activation="relu"))
model_2.add(BatchNormalization()) # Batch Normalization layer
model_2.add(Dense(32, activation='relu'))
model_2.add(BatchNormalization()) # Batch Normalization layer
model_2.add(Dense(1, activation='relu'))
optimizer = keras.optimizers.Adam(clipnorm=1.0) # Adam with gradient clipping
model_2.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy']) #trying accuracy
early_stopping = EarlyStopping(monitor='val_accuracy', patience=5, restore_best_weights=True) #stops if accuracy doenst improve after 5 epochs
start = time.time()
history = model_2.fit(X_train, y_train, validation_data=(X_val, y_val),
batch_size=100, epochs=25, callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/25
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
64/64 ━━━━━━━━━━━━━━━━━━━━ 0s 118ms/step - accuracy: 0.6761 - loss: 3.5233
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
64/64 ━━━━━━━━━━━━━━━━━━━━ 9s 125ms/step - accuracy: 0.6769 - loss: 3.5130 - val_accuracy: 0.8006 - val_loss: 2.9040 Epoch 2/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 123ms/step - accuracy: 0.8175 - loss: 1.8314 - val_accuracy: 0.8000 - val_loss: 2.7212 Epoch 3/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 123ms/step - accuracy: 0.8362 - loss: 1.5584 - val_accuracy: 0.8106 - val_loss: 1.9331 Epoch 4/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 124ms/step - accuracy: 0.8512 - loss: 1.2954 - val_accuracy: 0.8238 - val_loss: 1.6080 Epoch 5/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8605 - loss: 1.1375 - val_accuracy: 0.8219 - val_loss: 1.1258 Epoch 6/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 124ms/step - accuracy: 0.8554 - loss: 1.0733 - val_accuracy: 0.8256 - val_loss: 1.2252 Epoch 7/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 123ms/step - accuracy: 0.8646 - loss: 0.9337 - val_accuracy: 0.8400 - val_loss: 1.0676 Epoch 8/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8681 - loss: 0.8825 - val_accuracy: 0.8481 - val_loss: 1.1021 Epoch 9/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 124ms/step - accuracy: 0.8667 - loss: 0.8746 - val_accuracy: 0.8394 - val_loss: 0.9627 Epoch 10/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8699 - loss: 0.7232 - val_accuracy: 0.8350 - val_loss: 1.0835 Epoch 11/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8675 - loss: 0.6428 - val_accuracy: 0.8250 - val_loss: 0.8817 Epoch 12/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 122ms/step - accuracy: 0.8681 - loss: 0.6248 - val_accuracy: 0.8444 - val_loss: 0.8359 Epoch 13/25 64/64 ━━━━━━━━━━━━━━━━━━━━ 8s 125ms/step - accuracy: 0.8815 - loss: 0.5425 - val_accuracy: 0.8344 - val_loss: 0.8924 Time taken in seconds 103.79582643508911
now we will tune this adam one
plot(history,'loss')
plot(history,'accuracy')
recall is actually worse with more parameters here
val_pred = model_2.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
13/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step precision recall f1-score support 0.0 0.87 0.95 0.91 1274 1.0 0.69 0.46 0.55 326 accuracy 0.85 1600 macro avg 0.78 0.70 0.73 1600 weighted avg 0.84 0.85 0.84 1600
the model is paying too much attention to the majority class but its getting a decent balance between recall and precision on our target variable the model needs more tuning for sure
the class imbalance is still an issue despite the added and tuned parameters.
The class imbalance is influencing the model's learning process, causing it to prioritize the majority class even with tuned parameters and adaptive learning rates.
SGD with momentum performs better in scenarios with imbalanced datasets in many cases so we will try that first.
Neural Network with Balanced Data (by applying SMOTE) and SGD Optimizer¶
sm = SMOTE(random_state=1)
X_train_smote, y_train_smote= sm.fit_resample(X_train, y_train)
print('After UpSampling, the shape of train_X: {}'.format(X_train_smote.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train_smote.shape))
After UpSampling, the shape of train_X: (10192, 11) After UpSampling, the shape of train_y: (10192,)
# Scale the SMOTE-resampled data
scaler_smote = StandardScaler() # Create a new scaler for SMOTE data
X_train_smote_scaled = scaler_smote.fit_transform(X_train_smote)
tf.keras.backend.clear_session()
model_3 = Sequential()
model_3.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1]))
model_3.add(Dense(64, activation="relu"))
model_3.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_3.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
early_stopping = EarlyStopping(monitor='val_recall', patience=5, restore_best_weights=True)
start = time.time()
history = model_3.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
batch_size=32, epochs=100, callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.6750 - recall: 0.6456
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6749 - recall: 0.6457 - val_loss: 0.6121 - val_recall: 0.6442 Epoch 2/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5953 - recall: 0.7106 - val_loss: 0.5602 - val_recall: 0.6748 Epoch 3/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5564 - recall: 0.7139 - val_loss: 0.5393 - val_recall: 0.6963 Epoch 4/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5353 - recall: 0.7193 - val_loss: 0.5150 - val_recall: 0.6994 Epoch 5/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5152 - recall: 0.7187 - val_loss: 0.5251 - val_recall: 0.7607 Epoch 6/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4967 - recall: 0.7591 - val_loss: 0.5063 - val_recall: 0.7515 Epoch 7/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4818 - recall: 0.7617 - val_loss: 0.5162 - val_recall: 0.7791 Epoch 8/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4652 - recall: 0.7765 - val_loss: 0.5300 - val_recall: 0.7945 Epoch 9/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 13s 40ms/step - loss: 0.4530 - recall: 0.7935 - val_loss: 0.5347 - val_recall: 0.7914 Epoch 10/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 13s 39ms/step - loss: 0.4417 - recall: 0.7939 - val_loss: 0.5603 - val_recall: 0.8160 Epoch 11/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4355 - recall: 0.7979 - val_loss: 0.5510 - val_recall: 0.8098 Epoch 12/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4324 - recall: 0.7840 - val_loss: 0.5737 - val_recall: 0.8282 Epoch 13/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 13s 40ms/step - loss: 0.4237 - recall: 0.8001 - val_loss: 0.5697 - val_recall: 0.8129 Epoch 14/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4220 - recall: 0.7935 - val_loss: 0.5720 - val_recall: 0.8129 Epoch 15/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4201 - recall: 0.7958 - val_loss: 0.5881 - val_recall: 0.8252 Epoch 16/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4110 - recall: 0.8065 - val_loss: 0.6081 - val_recall: 0.8374 Epoch 17/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4143 - recall: 0.8028 - val_loss: 0.5953 - val_recall: 0.8344 Epoch 18/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4194 - recall: 0.7942 - val_loss: 0.5994 - val_recall: 0.8313 Epoch 19/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4056 - recall: 0.8047 - val_loss: 0.5826 - val_recall: 0.8190 Epoch 20/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4025 - recall: 0.7908 - val_loss: 0.6612 - val_recall: 0.8589 Epoch 21/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4061 - recall: 0.8095 - val_loss: 0.6422 - val_recall: 0.8344 Epoch 22/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4089 - recall: 0.8148 - val_loss: 0.6824 - val_recall: 0.8620 Epoch 23/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4017 - recall: 0.8150 - val_loss: 0.6773 - val_recall: 0.8620 Epoch 24/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4011 - recall: 0.8064 - val_loss: 0.6790 - val_recall: 0.8589 Epoch 25/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4002 - recall: 0.8009 - val_loss: 0.6869 - val_recall: 0.8589 Epoch 26/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.3918 - recall: 0.8169 - val_loss: 0.6989 - val_recall: 0.8558 Epoch 27/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.3812 - recall: 0.8175 - val_loss: 0.7176 - val_recall: 0.8589 Time taken in seconds 332.36059737205505
plot(history,'loss')
here the loss is still a major problem despite the smote fix we will have to do more tuning simply adding more epochs or adjusting the learning rates wont fix this...
plot(history,'recall')
this is much better but the oscilations are still a bit wild.... more epochs and a slower learning rate should be included in the next model. I think playing with the momentum may also be a good idea.
val_pred = model_3.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
40/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step precision recall f1-score support 0.0 0.94 0.57 0.71 1274 1.0 0.34 0.86 0.49 326 accuracy 0.63 1600 macro avg 0.64 0.72 0.60 1600 weighted avg 0.82 0.63 0.67 1600
our risk management department may or may not enjoy this... we amy want to switch to f1 so we can gett better precisioon
results.loc[3] = [2, "128,64", "relu,relu", 100, 32, "SGD", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer¶
model_4 = Sequential()
model_4.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1],
kernel_regularizer=regularizers.l2(0.01)))
model_4.add(BatchNormalization())
model_4.add(Dropout(0.5)) # Add dropout
model_4.add(Dense(64, activation="relu", kernel_regularizer=regularizers.l1(0.001)))
model_4.add(BatchNormalization())
model_4.add(Dropout(0.5)) # Add dropout
model_4.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.Adam()
model_4.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
early_stopping = EarlyStopping(monitor='val_recall', patience=5, restore_best_weights=True)
start = time.time()
history = model_4.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
batch_size=32, epochs=50,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 98ms/step - loss: 1.5942 - recall: 0.6297
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 1.5938 - recall: 0.6298 - val_loss: 1.2435 - val_recall: 0.7975 Epoch 2/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 1.1816 - recall: 0.7091 - val_loss: 1.0439 - val_recall: 0.8037 Epoch 3/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 0.9515 - recall: 0.7525 - val_loss: 0.9200 - val_recall: 0.8344 Epoch 4/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 105ms/step - loss: 0.8064 - recall: 0.7616 - val_loss: 0.8204 - val_recall: 0.8282 Epoch 5/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 0.7106 - recall: 0.7632 - val_loss: 0.7390 - val_recall: 0.8282 Epoch 6/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 0.6294 - recall: 0.7748 - val_loss: 0.6246 - val_recall: 0.7607 Epoch 7/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 0.6003 - recall: 0.7791 - val_loss: 0.6466 - val_recall: 0.7853 Epoch 8/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 102ms/step - loss: 0.5776 - recall: 0.7710 - val_loss: 0.6591 - val_recall: 0.7975 Time taken in seconds 264.18181252479553
plot(history,'loss')
plot(history,'recall')
val_pred = model_4.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
22/50 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step precision recall f1-score support 0.0 0.94 0.64 0.76 1274 1.0 0.37 0.83 0.51 326 accuracy 0.68 1600 macro avg 0.65 0.74 0.64 1600 weighted avg 0.82 0.68 0.71 1600
results.loc[4] = [2, "128,64", "relu,relu", 100, 32, "Adam", "0, -", "xavier", "L2,L1", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
SGD does indeed do better on imbalanced data at least in this case...
Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout¶
model_5 = Sequential()
model_5.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1],
kernel_regularizer=regularizers.l2(0.01)))
model_5.add(BatchNormalization())
model_5.add(Dropout(0.5))
model_5.add(Dense(64, activation="relu", kernel_regularizer=regularizers.l1(0.001)))
model_5.add(BatchNormalization())
model_5.add(Dropout(0.5))
model_5.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.Adam()
model_5.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
early_stopping = EarlyStopping(monitor='val_recall', patience=5, restore_best_weights=True)
start = time.time()
history = model_5.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
batch_size=32, epochs=100,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 99ms/step - loss: 1.6560 - recall: 0.6191
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 1.6555 - recall: 0.6192 - val_loss: 1.3356 - val_recall: 0.8712 Epoch 2/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 1.2473 - recall: 0.7112 - val_loss: 1.1217 - val_recall: 0.8589 Epoch 3/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 1.0160 - recall: 0.7429 - val_loss: 0.9498 - val_recall: 0.8129 Epoch 4/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 0.8482 - recall: 0.7444 - val_loss: 0.7906 - val_recall: 0.7638 Epoch 5/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 103ms/step - loss: 0.7272 - recall: 0.7484 - val_loss: 0.7154 - val_recall: 0.8160 Epoch 6/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 33s 104ms/step - loss: 0.6460 - recall: 0.7724 - val_loss: 0.6930 - val_recall: 0.8067 Time taken in seconds 198.04284501075745
plot(history,'loss')
plot(history,'recall')
val_pred = model_5.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
22/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step precision recall f1-score support 0.0 0.95 0.59 0.72 1274 1.0 0.35 0.87 0.50 326 accuracy 0.64 1600 macro avg 0.65 0.73 0.61 1600 weighted avg 0.83 0.64 0.68 1600
results.loc[5] = [2, "128,64", "relu,relu", 100, 32, "Adam", "0, -", "xavier", "L2,L1", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
this did not go as expected so it might be a good idea to try a smaller batch size
model_6 = Sequential()
model_6.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1],
kernel_regularizer=regularizers.l2(0.01))) # L2 regularization
model_6.add(BatchNormalization())
model_6.add(Dropout(0.5))
model_6.add(Dense(64, activation="relu",
kernel_regularizer=regularizers.l1(0.001))) # L1 regularization
model_6.add(BatchNormalization())
model_6.add(Dropout(0.5))
model_6.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.Adam()
model_6.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall']) #recall
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
start = time.time()
history = model_6.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
batch_size=20, epochs=50,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
510/510 ━━━━━━━━━━━━━━━━━━━━ 0s 99ms/step - loss: 1.6235 - recall: 0.6274
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 103ms/step - loss: 1.6232 - recall: 0.6275 - val_loss: 1.1850 - val_recall: 0.7914 Epoch 2/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 1.1569 - recall: 0.7094 - val_loss: 0.9852 - val_recall: 0.7945 Epoch 3/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 103ms/step - loss: 0.9041 - recall: 0.7510 - val_loss: 0.8212 - val_recall: 0.7914 Epoch 4/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 104ms/step - loss: 0.7563 - recall: 0.7525 - val_loss: 0.7445 - val_recall: 0.8067 Epoch 5/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 104ms/step - loss: 0.6671 - recall: 0.7568 - val_loss: 0.7064 - val_recall: 0.7975 Epoch 6/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.6168 - recall: 0.7636 - val_loss: 0.6678 - val_recall: 0.8098 Epoch 7/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.6013 - recall: 0.7584 - val_loss: 0.6084 - val_recall: 0.7730 Epoch 8/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.5857 - recall: 0.7566 - val_loss: 0.5993 - val_recall: 0.7577 Epoch 9/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 103ms/step - loss: 0.5750 - recall: 0.7612 - val_loss: 0.6284 - val_recall: 0.7822 Epoch 10/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.5627 - recall: 0.7661 - val_loss: 0.6188 - val_recall: 0.8221 Epoch 11/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 102ms/step - loss: 0.5582 - recall: 0.7689 - val_loss: 0.6378 - val_recall: 0.8098 Epoch 12/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 53s 103ms/step - loss: 0.5611 - recall: 0.7763 - val_loss: 0.6252 - val_recall: 0.7975 Epoch 13/50 510/510 ━━━━━━━━━━━━━━━━━━━━ 52s 103ms/step - loss: 0.5540 - recall: 0.7722 - val_loss: 0.6071 - val_recall: 0.7791 Time taken in seconds 681.8022449016571
plot(history,'loss')
plot(history,'recall')
val_pred = model_6.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
22/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step precision recall f1-score support 0.0 0.92 0.74 0.82 1274 1.0 0.43 0.76 0.55 326 accuracy 0.74 1600 macro avg 0.68 0.75 0.69 1600 weighted avg 0.82 0.74 0.77 1600
accuracy is better and so is precison of our target variable... but recall is slipping
results.loc[6] = [2, "128,64", "relu,relu", 50, 20, "Adam", "0, -", "xavier", "L2,L1", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
| 6 | 2 | 128,64 | relu,relu | 50 | 8 | Adam | 0, - | xavier | L2,L1 | 0.550212 | 0.607145 | 0.777473 | 0.779141 | 681.80 |
now we will go back to SGD becuase that handles the imbalance best and tweak our best model to get better performance
tf.keras.backend.clear_session()
model_7 = Sequential()
model_7.add(Dense(64, activation="tanh", input_dim=X_train_smote.shape[1])) #tanh
model_7.add(Dense(32, activation="tanh"))#tanh
model_7.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_7.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
start = time.time()
history = model_7.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
batch_size=32, epochs=50,
callbacks=[early_stopping])
end = time.time()
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.6419 - recall: 0.6238
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6418 - recall: 0.6240 - val_loss: 0.5573 - val_recall: 0.6626 Epoch 2/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5602 - recall: 0.7059 - val_loss: 0.5554 - val_recall: 0.6656 Epoch 3/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.5611 - recall: 0.7009 - val_loss: 0.5577 - val_recall: 0.7086 Epoch 4/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5560 - recall: 0.7080 - val_loss: 0.5593 - val_recall: 0.7147 Epoch 5/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5471 - recall: 0.7161 - val_loss: 0.5561 - val_recall: 0.7117 Epoch 6/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5515 - recall: 0.7139 - val_loss: 0.5370 - val_recall: 0.7086 Epoch 7/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5470 - recall: 0.7069 - val_loss: 0.5439 - val_recall: 0.7301 Epoch 8/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5338 - recall: 0.7310 - val_loss: 0.5389 - val_recall: 0.7515 Epoch 9/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5208 - recall: 0.7322 - val_loss: 0.5365 - val_recall: 0.7515 Epoch 10/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5116 - recall: 0.7561 - val_loss: 0.5211 - val_recall: 0.7607 Epoch 11/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5010 - recall: 0.7454 - val_loss: 0.5223 - val_recall: 0.7638 Epoch 12/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4951 - recall: 0.7643 - val_loss: 0.5172 - val_recall: 0.7761 Epoch 13/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4827 - recall: 0.7750 - val_loss: 0.5109 - val_recall: 0.7730 Epoch 14/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4726 - recall: 0.7720 - val_loss: 0.5247 - val_recall: 0.7975 Epoch 15/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4592 - recall: 0.7852 - val_loss: 0.5116 - val_recall: 0.7730 Epoch 16/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4670 - recall: 0.7714 - val_loss: 0.4973 - val_recall: 0.7761 Epoch 17/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4560 - recall: 0.7743 - val_loss: 0.5017 - val_recall: 0.7638 Epoch 18/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4523 - recall: 0.7876 - val_loss: 0.5096 - val_recall: 0.7883 Epoch 19/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4520 - recall: 0.7760 - val_loss: 0.4970 - val_recall: 0.7730 Epoch 20/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4465 - recall: 0.7754 - val_loss: 0.5119 - val_recall: 0.7883 Epoch 21/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4426 - recall: 0.7831 - val_loss: 0.4983 - val_recall: 0.7730 Epoch 22/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4380 - recall: 0.7832 - val_loss: 0.5039 - val_recall: 0.7761 Epoch 23/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4282 - recall: 0.7984 - val_loss: 0.4962 - val_recall: 0.7699 Epoch 24/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4399 - recall: 0.7903 - val_loss: 0.5092 - val_recall: 0.7791 Epoch 25/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4222 - recall: 0.8007 - val_loss: 0.4948 - val_recall: 0.7546 Epoch 26/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4321 - recall: 0.7908 - val_loss: 0.4955 - val_recall: 0.7638 Epoch 27/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4200 - recall: 0.7906 - val_loss: 0.4961 - val_recall: 0.7577 Epoch 28/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4233 - recall: 0.8025 - val_loss: 0.4897 - val_recall: 0.7423 Epoch 29/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4226 - recall: 0.7962 - val_loss: 0.4897 - val_recall: 0.7515 Epoch 30/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4197 - recall: 0.7955 - val_loss: 0.4999 - val_recall: 0.7577 Epoch 31/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4108 - recall: 0.8076 - val_loss: 0.5041 - val_recall: 0.7669 Epoch 32/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4203 - recall: 0.7968 - val_loss: 0.4935 - val_recall: 0.7515 Epoch 33/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4160 - recall: 0.8049 - val_loss: 0.4926 - val_recall: 0.7362
plot(history,'loss')
plot(history,'recall')
val_pred = model_7.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
40/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step precision recall f1-score support 0.0 0.92 0.77 0.84 1274 1.0 0.45 0.74 0.56 326 accuracy 0.76 1600 macro avg 0.69 0.75 0.70 1600 weighted avg 0.82 0.76 0.78 1600
results.loc[7] = [2, "64,32", "tanh,tanh", 50, 32, "SGD", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
| 6 | 2 | 128,64 | relu,relu | 50 | 8 | Adam | 0, - | xavier | L2,L1 | 0.550212 | 0.607145 | 0.777473 | 0.779141 | 681.80 |
| 7 | 2 | 64,32 | tanh,tanh | 50 | 32 | SGD | 0.001, - | xavier | - | 0.417627 | 0.492607 | 0.799843 | 0.736196 | 404.54 |
tf.keras.backend.clear_session()
model_8 = Sequential()
model_8.add(Dense(32, activation="sigmoid", input_dim=X_train_smote.shape[1])) #sigmoid
model_8.add(Dense(16, activation="sigmoid")) #sigmoid
model_8.add(Dense(1, activation='sigmoid')) #sigmoid
optimizer = tf.keras.optimizers.SGD(learning_rate=0.0001, momentum=0.9) #low learning rate
model_8.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
early_stopping = EarlyStopping(monitor='val_recall', patience=5, restore_best_weights=True)
start = time.time()
history = model_8.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
batch_size=32, epochs=50,
callbacks=[early_stopping])
end = time.time()
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.7095 - recall: 0.0000e+00
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.7095 - recall: 0.0000e+00 - val_loss: 0.6368 - val_recall: 0.0000e+00 Epoch 2/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6992 - recall: 0.0028 - val_loss: 0.6624 - val_recall: 0.0245 Epoch 3/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6955 - recall: 0.0374 - val_loss: 0.6772 - val_recall: 0.1319 Epoch 4/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6941 - recall: 0.1626 - val_loss: 0.6862 - val_recall: 0.2577 Epoch 5/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6933 - recall: 0.3287 - val_loss: 0.6916 - val_recall: 0.3681 Epoch 6/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6932 - recall: 0.4331 - val_loss: 0.6949 - val_recall: 0.4724 Epoch 7/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6924 - recall: 0.5261 - val_loss: 0.6969 - val_recall: 0.5368 Epoch 8/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6917 - recall: 0.5781 - val_loss: 0.6977 - val_recall: 0.5798 Epoch 9/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6918 - recall: 0.6352 - val_loss: 0.6979 - val_recall: 0.5951 Epoch 10/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6919 - recall: 0.5989 - val_loss: 0.6984 - val_recall: 0.6074 Epoch 11/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6911 - recall: 0.6401 - val_loss: 0.6978 - val_recall: 0.6104 Epoch 12/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6910 - recall: 0.6598 - val_loss: 0.6972 - val_recall: 0.6074 Epoch 13/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6899 - recall: 0.6381 - val_loss: 0.6972 - val_recall: 0.6196 Epoch 14/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6899 - recall: 0.6526 - val_loss: 0.6970 - val_recall: 0.6258 Epoch 15/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6901 - recall: 0.6726 - val_loss: 0.6966 - val_recall: 0.6288 Epoch 16/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6893 - recall: 0.6541 - val_loss: 0.6967 - val_recall: 0.6442 Epoch 17/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6887 - recall: 0.6772 - val_loss: 0.6952 - val_recall: 0.6288 Epoch 18/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6888 - recall: 0.6718 - val_loss: 0.6946 - val_recall: 0.6258 Epoch 19/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6882 - recall: 0.6307 - val_loss: 0.6958 - val_recall: 0.6687 Epoch 20/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6876 - recall: 0.6997 - val_loss: 0.6954 - val_recall: 0.6718 Epoch 21/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6876 - recall: 0.6828 - val_loss: 0.6951 - val_recall: 0.6810 Epoch 22/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6872 - recall: 0.6730 - val_loss: 0.6945 - val_recall: 0.6748 Epoch 23/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6865 - recall: 0.6970 - val_loss: 0.6942 - val_recall: 0.6840 Epoch 24/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6867 - recall: 0.6705 - val_loss: 0.6934 - val_recall: 0.6687 Epoch 25/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6861 - recall: 0.6740 - val_loss: 0.6936 - val_recall: 0.6963 Epoch 26/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6859 - recall: 0.6716 - val_loss: 0.6931 - val_recall: 0.6902 Epoch 27/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6852 - recall: 0.6955 - val_loss: 0.6922 - val_recall: 0.6748 Epoch 28/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6847 - recall: 0.6997 - val_loss: 0.6923 - val_recall: 0.6933 Epoch 29/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6849 - recall: 0.6770 - val_loss: 0.6921 - val_recall: 0.6963 Epoch 30/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6841 - recall: 0.6901 - val_loss: 0.6921 - val_recall: 0.7117 Epoch 31/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6837 - recall: 0.7015 - val_loss: 0.6917 - val_recall: 0.7117 Epoch 32/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6828 - recall: 0.6971 - val_loss: 0.6915 - val_recall: 0.7178 Epoch 33/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6826 - recall: 0.7202 - val_loss: 0.6910 - val_recall: 0.7055 Epoch 34/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6822 - recall: 0.6846 - val_loss: 0.6915 - val_recall: 0.7423 Epoch 35/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6823 - recall: 0.7073 - val_loss: 0.6916 - val_recall: 0.7515 Epoch 36/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6819 - recall: 0.7012 - val_loss: 0.6912 - val_recall: 0.7423 Epoch 37/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6813 - recall: 0.7241 - val_loss: 0.6905 - val_recall: 0.7270 Epoch 38/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 13s 39ms/step - loss: 0.6809 - recall: 0.7082 - val_loss: 0.6900 - val_recall: 0.7239 Epoch 39/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6809 - recall: 0.7052 - val_loss: 0.6894 - val_recall: 0.7178 Epoch 40/50 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6805 - recall: 0.6964 - val_loss: 0.6893 - val_recall: 0.7239
plot(history,'loss')
plot(history,'recall')
val_pred = model_8.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
39/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step precision recall f1-score support 0.0 0.88 0.46 0.60 1274 1.0 0.26 0.75 0.39 326 accuracy 0.52 1600 macro avg 0.57 0.60 0.49 1600 weighted avg 0.75 0.52 0.56 1600
results.loc[8] = [2, "32,16", "sigmoid,sigmoid", 50, 32, "SGD", "0.0001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
| 6 | 2 | 128,64 | relu,relu | 50 | 8 | Adam | 0, - | xavier | L2,L1 | 0.550212 | 0.607145 | 0.777473 | 0.779141 | 681.80 |
| 7 | 2 | 64,32 | tanh,tanh | 50 | 32 | SGD | 0.001, - | xavier | - | 0.417627 | 0.492607 | 0.799843 | 0.736196 | 404.54 |
| 8 | 2 | 32,16 | sigmoid,sigmoid | 50 | 32 | SGD | 0.0001, - | xavier | - | 0.680181 | 0.689308 | 0.696625 | 0.723926 | 490.79 |
tf.keras.backend.clear_session()
model_9 = Sequential()
model_9.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1]))
model_9.add(Dense(64, activation="relu"))
model_9.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_9.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
start = time.time()
history = model_9.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
batch_size=32, epochs=100,
callbacks=[early_stopping]) # Pass the callback to fit
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.6697 - recall: 0.8128
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.6696 - recall: 0.8126 - val_loss: 0.6018 - val_recall: 0.6288 Epoch 2/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.6014 - recall: 0.6672 - val_loss: 0.5683 - val_recall: 0.6963 Epoch 3/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.5607 - recall: 0.6919 - val_loss: 0.5514 - val_recall: 0.7239 Epoch 4/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.5383 - recall: 0.7157 - val_loss: 0.5463 - val_recall: 0.7546 Epoch 5/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.5244 - recall: 0.7325 - val_loss: 0.5214 - val_recall: 0.7423 Epoch 6/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.5013 - recall: 0.7385 - val_loss: 0.5166 - val_recall: 0.7638 Epoch 7/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4806 - recall: 0.7484 - val_loss: 0.5325 - val_recall: 0.7853 Epoch 8/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4718 - recall: 0.7604 - val_loss: 0.5369 - val_recall: 0.7945 Epoch 9/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 38ms/step - loss: 0.4568 - recall: 0.7671 - val_loss: 0.5494 - val_recall: 0.8098 Epoch 10/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4447 - recall: 0.7776 - val_loss: 0.5659 - val_recall: 0.8190 Epoch 11/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 12s 39ms/step - loss: 0.4364 - recall: 0.7866 - val_loss: 0.5565 - val_recall: 0.8067 Time taken in seconds 135.95033502578735
plot(history,'loss')
plot(history,'recall')
val_pred = model_9.predict(X_val)
val_pred = (val_pred > 0.5)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
39/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step precision recall f1-score support 0.0 0.92 0.74 0.82 1274 1.0 0.43 0.76 0.55 326 accuracy 0.75 1600 macro avg 0.68 0.75 0.69 1600 weighted avg 0.82 0.75 0.77 1600
this is fairly balanced i kind of like it its not prefect but its no overly biased towards recall
results.loc[9] = [2, "128,64", "relu,relu", 150, 32, "SGD", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
| 6 | 2 | 128,64 | relu,relu | 50 | 8 | Adam | 0, - | xavier | L2,L1 | 0.550212 | 0.607145 | 0.777473 | 0.779141 | 681.80 |
| 7 | 2 | 64,32 | tanh,tanh | 50 | 32 | SGD | 0.001, - | xavier | - | 0.417627 | 0.492607 | 0.799843 | 0.736196 | 404.54 |
| 8 | 2 | 32,16 | sigmoid,sigmoid | 50 | 32 | SGD | 0.0001, - | xavier | - | 0.680181 | 0.689308 | 0.696625 | 0.723926 | 490.79 |
| 9 | 2 | 128,64 | relu,relu | 150 | 32 | SGD | 0.001, - | xavier | - | 0.434032 | 0.556475 | 0.788265 | 0.806748 | 135.95 |
we need to give the model more epochs and more momentum and a small batch size
tf.keras.backend.clear_session()
model_10 = Sequential()
model_10.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1]))
model_10.add(Dense(64, activation="relu"))
model_10.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.999) #larger momentum
model_10.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
start = time.time()
history = model_10.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
batch_size=16, epochs=100, #smaller batch size
)
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step - loss: 0.6045 - recall: 0.6807
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6044 - recall: 0.6807 - val_loss: 0.7312 - val_recall: 0.8374 Epoch 2/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4725 - recall: 0.7751 - val_loss: 0.5062 - val_recall: 0.7055 Epoch 3/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4443 - recall: 0.7705 - val_loss: 0.8109 - val_recall: 0.8742 Epoch 4/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4327 - recall: 0.8315 - val_loss: 0.6591 - val_recall: 0.8282 Epoch 5/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4140 - recall: 0.7881 - val_loss: 0.6423 - val_recall: 0.8681 Epoch 6/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4434 - recall: 0.8479 - val_loss: 0.6399 - val_recall: 0.8497 Epoch 7/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4276 - recall: 0.8049 - val_loss: 0.6787 - val_recall: 0.7822 Epoch 8/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4205 - recall: 0.7256 - val_loss: 0.7436 - val_recall: 0.9080 Epoch 9/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4239 - recall: 0.8807 - val_loss: 0.9416 - val_recall: 0.7699 Epoch 10/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4299 - recall: 0.6948 - val_loss: 0.6749 - val_recall: 0.7117 Epoch 11/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4057 - recall: 0.7039 - val_loss: 0.6068 - val_recall: 0.7975 Epoch 12/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4152 - recall: 0.7237 - val_loss: 0.7627 - val_recall: 0.8252 Epoch 13/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.3938 - recall: 0.7537 - val_loss: 1.0532 - val_recall: 0.8282 Epoch 14/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4114 - recall: 0.7439 - val_loss: 0.9321 - val_recall: 0.9325 Epoch 15/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4185 - recall: 0.9097 - val_loss: 0.6232 - val_recall: 0.7914 Epoch 16/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4302 - recall: 0.7005 - val_loss: 0.7076 - val_recall: 0.9233 Epoch 17/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4428 - recall: 0.9022 - val_loss: 1.0876 - val_recall: 0.9387 Epoch 18/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.4560 - recall: 0.9216 - val_loss: 1.1865 - val_recall: 0.9724 Epoch 19/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4752 - recall: 0.8886 - val_loss: 0.8114 - val_recall: 0.9724 Epoch 20/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.5016 - recall: 0.9049 - val_loss: 0.8612 - val_recall: 0.9632 Epoch 21/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4948 - recall: 0.8929 - val_loss: 0.8160 - val_recall: 0.9264 Epoch 22/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.4903 - recall: 0.8770 - val_loss: 0.5357 - val_recall: 0.1288 Epoch 23/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.5353 - recall: 0.7547 - val_loss: 0.7762 - val_recall: 0.9479 Epoch 24/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6011 - recall: 0.8326 - val_loss: 0.7171 - val_recall: 0.9693 Epoch 25/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.5854 - recall: 0.8631 - val_loss: 1.0178 - val_recall: 0.8313 Epoch 26/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.5475 - recall: 0.6876 - val_loss: 1.2611 - val_recall: 0.7638 Epoch 27/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.5512 - recall: 0.6260 - val_loss: 1.3986 - val_recall: 0.7577 Epoch 28/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6342 - recall: 0.5654 - val_loss: 1.5879 - val_recall: 0.4724 Epoch 29/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6160 - recall: 0.3916 - val_loss: 0.5662 - val_recall: 0.1043 Epoch 30/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6361 - recall: 0.5912 - val_loss: 0.8483 - val_recall: 0.9877 Epoch 31/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6178 - recall: 0.4242 - val_loss: 0.7379 - val_recall: 0.9540 Epoch 32/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6735 - recall: 0.9506 - val_loss: 0.6412 - val_recall: 0.9847 Epoch 33/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6642 - recall: 0.8124 - val_loss: 0.6228 - val_recall: 0.9417 Epoch 34/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6676 - recall: 0.6253 - val_loss: 0.6481 - val_recall: 0.9356 Epoch 35/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6406 - recall: 0.9130 - val_loss: 0.6469 - val_recall: 0.9663 Epoch 36/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6392 - recall: 0.6713 - val_loss: 0.6946 - val_recall: 0.9755 Epoch 37/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6433 - recall: 0.9738 - val_loss: 0.7107 - val_recall: 0.9632 Epoch 38/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6327 - recall: 0.9615 - val_loss: 0.6083 - val_recall: 0.0092 Epoch 39/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 38ms/step - loss: 0.6663 - recall: 0.7649 - val_loss: 0.6283 - val_recall: 0.9939 Epoch 40/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6404 - recall: 0.9860 - val_loss: 0.6256 - val_recall: 0.9847 Epoch 41/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6334 - recall: 0.9725 - val_loss: 0.6813 - val_recall: 0.9755 Epoch 42/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6538 - recall: 0.9128 - val_loss: 0.7284 - val_recall: 0.9969 Epoch 43/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6911 - recall: 0.6940 - val_loss: 0.6519 - val_recall: 0.0031 Epoch 44/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6818 - recall: 0.5046 - val_loss: 0.7775 - val_recall: 0.9939 Epoch 45/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6699 - recall: 0.6966 - val_loss: 0.6893 - val_recall: 0.9847 Epoch 46/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6728 - recall: 0.9932 - val_loss: 0.6783 - val_recall: 0.9755 Epoch 47/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6515 - recall: 0.9215 - val_loss: 0.7836 - val_recall: 0.9785 Epoch 48/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6500 - recall: 0.8488 - val_loss: 0.6215 - val_recall: 0.9417 Epoch 49/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6213 - recall: 0.9765 - val_loss: 0.9208 - val_recall: 0.9908 Epoch 50/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6576 - recall: 0.7119 - val_loss: 0.6144 - val_recall: 0.0000e+00 Epoch 51/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6897 - recall: 0.3887 - val_loss: 0.8826 - val_recall: 0.9939 Epoch 52/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6868 - recall: 0.6323 - val_loss: 0.6340 - val_recall: 0.0000e+00 Epoch 53/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6606 - recall: 0.6389 - val_loss: 0.7373 - val_recall: 0.9785 Epoch 54/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6393 - recall: 0.9858 - val_loss: 0.6918 - val_recall: 0.9663 Epoch 55/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6472 - recall: 0.9193 - val_loss: 0.7436 - val_recall: 0.9816 Epoch 56/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6243 - recall: 0.8104 - val_loss: 0.5900 - val_recall: 0.9417 Epoch 57/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6413 - recall: 0.9114 - val_loss: 0.7386 - val_recall: 0.9785 Epoch 58/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6537 - recall: 0.8111 - val_loss: 0.6396 - val_recall: 0.9785 Epoch 59/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6557 - recall: 0.3432 - val_loss: 0.6838 - val_recall: 0.9816 Epoch 60/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6380 - recall: 0.8517 - val_loss: 0.8151 - val_recall: 0.9847 Epoch 61/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6135 - recall: 0.4520 - val_loss: 0.7738 - val_recall: 0.9755 Epoch 62/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6142 - recall: 0.8386 - val_loss: 0.7250 - val_recall: 0.9755 Epoch 63/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6311 - recall: 0.5310 - val_loss: 0.8438 - val_recall: 0.9663 Epoch 64/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6125 - recall: 0.9784 - val_loss: 0.8696 - val_recall: 0.9540 Epoch 65/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6045 - recall: 0.9431 - val_loss: 0.9773 - val_recall: 0.9356 Epoch 66/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6360 - recall: 0.9299 - val_loss: 1.1951 - val_recall: 0.9724 Epoch 67/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6992 - recall: 0.9979 - val_loss: 1.0997 - val_recall: 0.9755 Epoch 68/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6199 - recall: 0.9883 - val_loss: 0.9158 - val_recall: 0.9663 Epoch 69/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6254 - recall: 0.9857 - val_loss: 0.7324 - val_recall: 0.9693 Epoch 70/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6085 - recall: 0.9934 - val_loss: 0.8071 - val_recall: 0.9724 Epoch 71/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6170 - recall: 0.9935 - val_loss: 0.8005 - val_recall: 0.9724 Epoch 72/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6212 - recall: 0.9932 - val_loss: 0.7184 - val_recall: 0.9663 Epoch 73/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 38ms/step - loss: 0.6190 - recall: 0.6870 - val_loss: 0.8518 - val_recall: 0.9693 Epoch 74/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6153 - recall: 0.9926 - val_loss: 0.8498 - val_recall: 0.0000e+00 Epoch 75/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6787 - recall: 0.7039 - val_loss: 1.0993 - val_recall: 0.9755 Epoch 76/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6475 - recall: 0.6497 - val_loss: 0.7246 - val_recall: 0.0000e+00 Epoch 77/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6609 - recall: 0.5842 - val_loss: 0.9813 - val_recall: 0.9847 Epoch 78/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6537 - recall: 0.7260 - val_loss: 0.6909 - val_recall: 0.0000e+00 Epoch 79/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6576 - recall: 0.6112 - val_loss: 0.7906 - val_recall: 0.9877 Epoch 80/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6542 - recall: 0.8324 - val_loss: 0.6328 - val_recall: 0.0000e+00 Epoch 81/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6538 - recall: 0.6453 - val_loss: 0.7568 - val_recall: 0.9877 Epoch 82/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6277 - recall: 0.9968 - val_loss: 0.6605 - val_recall: 0.9663 Epoch 83/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6223 - recall: 0.9933 - val_loss: 0.7700 - val_recall: 0.9785 Epoch 84/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6191 - recall: 0.9967 - val_loss: 0.7342 - val_recall: 0.9785 Epoch 85/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 25s 39ms/step - loss: 0.6125 - recall: 0.9795 - val_loss: 0.7667 - val_recall: 0.9816 Epoch 86/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6243 - recall: 0.9983 - val_loss: 0.6680 - val_recall: 0.9847 Epoch 87/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6329 - recall: 1.0000 - val_loss: 0.7957 - val_recall: 0.9816 Epoch 88/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6316 - recall: 0.9667 - val_loss: 1.0267 - val_recall: 0.9816 Epoch 89/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6393 - recall: 0.8865 - val_loss: 0.7234 - val_recall: 0.9785 Epoch 90/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6481 - recall: 0.5827 - val_loss: 0.8466 - val_recall: 0.9847 Epoch 91/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6564 - recall: 0.7933 - val_loss: 0.7983 - val_recall: 0.0000e+00 Epoch 92/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6536 - recall: 0.4721 - val_loss: 1.0292 - val_recall: 0.9847 Epoch 93/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6260 - recall: 0.8216 - val_loss: 0.6948 - val_recall: 0.9755 Epoch 94/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6094 - recall: 0.9974 - val_loss: 0.9607 - val_recall: 0.9755 Epoch 95/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6315 - recall: 0.9969 - val_loss: 0.7830 - val_recall: 0.9816 Epoch 96/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6338 - recall: 0.9939 - val_loss: 0.7495 - val_recall: 0.0000e+00 Epoch 97/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6284 - recall: 0.6500 - val_loss: 0.8744 - val_recall: 0.9724 Epoch 98/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6140 - recall: 0.9968 - val_loss: 0.8675 - val_recall: 0.9693 Epoch 99/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6006 - recall: 0.9937 - val_loss: 0.8863 - val_recall: 0.9571 Epoch 100/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 24s 38ms/step - loss: 0.6087 - recall: 0.9953 - val_loss: 0.8465 - val_recall: 0.9571 Time taken in seconds 2449.964891910553
that was too many epochs we need more epoch but not that many.... maybe early stopping of 15 will do it or a max of 50 epochs
plot(history,'loss')
plot(history,'recall')
this is very interesting but i have no idea how interpret this....
val_pred = model_10.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
37/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step precision recall f1-score support 0.0 0.97 0.32 0.49 1274 1.0 0.27 0.96 0.42 326 accuracy 0.45 1600 macro avg 0.62 0.64 0.45 1600 weighted avg 0.82 0.45 0.47 1600
results.loc[10] = [2, "128,64", "relu,relu", 100, 16, "SGD", "0.001, -", "xavier", "-", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
| 6 | 2 | 128,64 | relu,relu | 50 | 8 | Adam | 0, - | xavier | L2,L1 | 0.550212 | 0.607145 | 0.777473 | 0.779141 | 681.80 |
| 7 | 2 | 64,32 | tanh,tanh | 50 | 32 | SGD | 0.001, - | xavier | - | 0.417627 | 0.492607 | 0.799843 | 0.736196 | 404.54 |
| 8 | 2 | 32,16 | sigmoid,sigmoid | 50 | 32 | SGD | 0.0001, - | xavier | - | 0.680181 | 0.689308 | 0.696625 | 0.723926 | 490.79 |
| 9 | 2 | 128,64 | relu,relu | 150 | 32 | SGD | 0.001, - | xavier | - | 0.434032 | 0.556475 | 0.788265 | 0.806748 | 135.95 |
| 10 | 2 | 128,64 | relu,relu | 100 | 16 | SGD | 0.001, - | xavier | - | 0.598579 | 0.846530 | 0.994309 | 0.957055 | 2449.96 |
our precision f1 scores are bad our recall is good but we need to see if we can get a model that would create better resource allocation for when it comes to our strategies to deal with churning
tf.keras.backend.clear_session()
model_11 = Sequential()
# Input layer with L2 regularization and Glorot uniform initialization
model_11.add(Dense(128, activation="relu", input_dim=X_train_smote.shape[1],
kernel_initializer='glorot_uniform', # Xavier Glorot
kernel_regularizer=regularizers.l2(0.001)))
model_11.add(BatchNormalization())
model_11.add(Dropout(0.5)) #dropout
model_11.add(Dense(64, activation="relu", kernel_initializer='glorot_uniform')) # Xavier Glorot
model_11.add(BatchNormalization())
model_11.add(Dropout(0.5)) #dropout
model_11.add(Dense(1, activation='sigmoid', kernel_initializer='glorot_uniform')) # Xavier Glorot
# Calculate class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.99)
model_11.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
#EarlyStopping patience=15 but on loss function
early_stopping = EarlyStopping(monitor='val_loss', patience=15, restore_best_weights=True)
start = time.time()
history = model_11.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=16, epochs=50,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 68ms/step - loss: 0.9731 - recall: 0.8383
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.9729 - recall: 0.8384 - val_loss: 1.2114 - val_recall: 0.9755 Epoch 2/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6950 - recall: 0.9488 - val_loss: 1.1756 - val_recall: 0.9632 Epoch 3/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6452 - recall: 0.9465 - val_loss: 1.1012 - val_recall: 0.9540 Epoch 4/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6397 - recall: 0.9455 - val_loss: 1.0429 - val_recall: 0.9724 Epoch 5/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6272 - recall: 0.9449 - val_loss: 0.9858 - val_recall: 0.9755 Epoch 6/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.6187 - recall: 0.9460 - val_loss: 1.1074 - val_recall: 0.9663 Epoch 7/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 47s 73ms/step - loss: 0.6091 - recall: 0.9431 - val_loss: 1.0829 - val_recall: 0.9724 Epoch 8/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5900 - recall: 0.9473 - val_loss: 1.0255 - val_recall: 0.9571 Epoch 9/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5932 - recall: 0.9496 - val_loss: 1.1489 - val_recall: 0.9755 Epoch 10/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5877 - recall: 0.9429 - val_loss: 1.1258 - val_recall: 0.9724 Epoch 11/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5837 - recall: 0.9487 - val_loss: 1.0471 - val_recall: 0.9601 Epoch 12/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 73ms/step - loss: 0.5732 - recall: 0.9485 - val_loss: 1.1743 - val_recall: 0.9785 Epoch 13/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 47s 73ms/step - loss: 0.5703 - recall: 0.9538 - val_loss: 1.0951 - val_recall: 0.9785 Epoch 14/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5864 - recall: 0.9617 - val_loss: 1.1145 - val_recall: 0.9785 Epoch 15/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5777 - recall: 0.9554 - val_loss: 1.0810 - val_recall: 0.9693 Epoch 16/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5665 - recall: 0.9500 - val_loss: 1.0420 - val_recall: 0.9509 Epoch 17/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5720 - recall: 0.9494 - val_loss: 1.0614 - val_recall: 0.9663 Epoch 18/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5784 - recall: 0.9490 - val_loss: 1.0778 - val_recall: 0.9693 Epoch 19/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5736 - recall: 0.9514 - val_loss: 1.1204 - val_recall: 0.9724 Epoch 20/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 46s 72ms/step - loss: 0.5814 - recall: 0.9475 - val_loss: 1.2318 - val_recall: 0.9693 Time taken in seconds 922.238805770874
with glorot we may be able to speed things up by bringing back early stopping with patience 10 or even 5!
plot(history,'loss')
plot(history,'recall')
val_pred = model_11.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
22/50 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step precision recall f1-score support 0.0 0.98 0.32 0.48 1274 1.0 0.27 0.98 0.42 326 accuracy 0.45 1600 macro avg 0.62 0.65 0.45 1600 weighted avg 0.84 0.45 0.47 1600
so far in our best model we are still predicting a lot of people will churn that will not that we will be wasting resources trying to stop from churning despite all these fancy hyperparameters the precision and accuracy are low
results.loc[11] = [2, "128,64", "relu,relu", 50, 16, "SGD", "0.001, -", "xavier", "L2,N/A", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
| 6 | 2 | 128,64 | relu,relu | 50 | 8 | Adam | 0, - | xavier | L2,L1 | 0.550212 | 0.607145 | 0.777473 | 0.779141 | 681.80 |
| 7 | 2 | 64,32 | tanh,tanh | 50 | 32 | SGD | 0.001, - | xavier | - | 0.417627 | 0.492607 | 0.799843 | 0.736196 | 404.54 |
| 8 | 2 | 32,16 | sigmoid,sigmoid | 50 | 32 | SGD | 0.0001, - | xavier | - | 0.680181 | 0.689308 | 0.696625 | 0.723926 | 490.79 |
| 9 | 2 | 128,64 | relu,relu | 150 | 32 | SGD | 0.001, - | xavier | - | 0.434032 | 0.556475 | 0.788265 | 0.806748 | 135.95 |
| 10 | 2 | 128,64 | relu,relu | 100 | 16 | SGD | 0.001, - | xavier | - | 0.598579 | 0.846530 | 0.994309 | 0.957055 | 2449.96 |
| 11 | 2 | 128,64 | relu,relu | 50 | 16 | SGD | 0.001, - | xavier | L2,N/A | 0.575172 | 1.231755 | 0.947214 | 0.969325 | 922.24 |
this is still better but the precision and f1 remain very low.
im going to lower the kernal regulizer parameter so that the model can hopefully capture more complexity.
also, relu can sometimes lead to "dead neurons" which can hinder learning. it would be a good idea to try LeakyReLU or ELU.
we could increase model complexity CAREFULLY
OR HIGHER CLASS WEIGHTS
lastly it may be a good idea to try He initilaizer
from tensorflow.keras.layers import LeakyReLU
tf.keras.backend.clear_session()
model_12 = Sequential()
model_12.add(Dense(128, activation=LeakyReLU(alpha=0.1),
input_dim=X_train_smote.shape[1],
kernel_initializer='glorot_uniform', # Xavier Glorot
kernel_regularizer=regularizers.l2(0.0001))) # L2 regularization
model_12.add(BatchNormalization()) # Batch Normalization
model_12.add(Dropout(0.5)) # Dropout
# Hidden layer
model_12.add(Dense(64, activation=LeakyReLU(alpha=0.1),
kernel_initializer='glorot_uniform', # Xavier Glorot
kernel_regularizer=regularizers.l2(0.0001))) # L2 regularization
model_12.add(BatchNormalization()) # Batch Normalization
model_12.add(Dropout(0.5)) # Dropout
model_12.add(Dense(32, activation='relu', kernel_initializer='glorot_uniform'))
model_12.add(Dense(1, activation='sigmoid', kernel_initializer='glorot_uniform'))
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_12.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
# Add EarlyStopping callback with patience=5
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
start = time.time()
history = model_12.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=32, epochs=100,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn( /usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 75ms/step - loss: 0.9232 - recall: 0.8638
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.9228 - recall: 0.8640 - val_loss: 0.9511 - val_recall: 0.9877 Epoch 2/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.7133 - recall: 0.9685 - val_loss: 0.9980 - val_recall: 0.9785 Epoch 3/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.6824 - recall: 0.9675 - val_loss: 0.9633 - val_recall: 0.9663 Epoch 4/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.6730 - recall: 0.9663 - val_loss: 0.9821 - val_recall: 0.9693 Epoch 5/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.6571 - recall: 0.9668 - val_loss: 0.9764 - val_recall: 0.9632 Epoch 6/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 25s 79ms/step - loss: 0.6575 - recall: 0.9629 - val_loss: 0.9730 - val_recall: 0.9601 Time taken in seconds 151.72719502449036
plot(history,'loss')
plot(history,'recall')
val_pred = model_12.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
13/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step precision recall f1-score support 0.0 0.96 0.07 0.13 1274 1.0 0.21 0.99 0.35 326 accuracy 0.26 1600 macro avg 0.59 0.53 0.24 1600 weighted avg 0.81 0.26 0.18 1600
recall was not impacted but our preeciion on target variable and the accuracy are worse
results.loc[12] = [3, "128,64,32", "leakyrelu,leakyrelu,relu", 100, 32, "SGD", "0.001, -", "xavier", "L2,L2,N/A", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
| 6 | 2 | 128,64 | relu,relu | 50 | 8 | Adam | 0, - | xavier | L2,L1 | 0.550212 | 0.607145 | 0.777473 | 0.779141 | 681.80 |
| 7 | 2 | 64,32 | tanh,tanh | 50 | 32 | SGD | 0.001, - | xavier | - | 0.417627 | 0.492607 | 0.799843 | 0.736196 | 404.54 |
| 8 | 2 | 32,16 | sigmoid,sigmoid | 50 | 32 | SGD | 0.0001, - | xavier | - | 0.680181 | 0.689308 | 0.696625 | 0.723926 | 490.79 |
| 9 | 2 | 128,64 | relu,relu | 150 | 32 | SGD | 0.001, - | xavier | - | 0.434032 | 0.556475 | 0.788265 | 0.806748 | 135.95 |
| 10 | 2 | 128,64 | relu,relu | 100 | 16 | SGD | 0.001, - | xavier | - | 0.598579 | 0.846530 | 0.994309 | 0.957055 | 2449.96 |
| 11 | 2 | 128,64 | relu,relu | 50 | 16 | SGD | 0.001, - | xavier | L2,N/A | 0.575172 | 1.231755 | 0.947214 | 0.969325 | 922.24 |
| 12 | 3 | 128,64,32 | leakyrelu,leakyrelu,relu | 100 | 32 | SGD | 0.001, - | xavier | L2,L2,N/A | 0.643929 | 0.973020 | 0.965463 | 0.960123 | 151.73 |
add more agressive class weights
tf.keras.backend.clear_session()
model_13 = Sequential()
model_13.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train_smote.shape[1],
kernel_regularizer=regularizers.l2(0.0001)))
model_13.add(Dropout(0.2))
model_13.add(Dense(64, activation=LeakyReLU(alpha=0.1)))
model_13.add(Dropout(0.2))
model_13.add(Dense(32, activation='relu'))
model_13.add(Dense(1, activation='sigmoid'))
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]} #bit more aggressive
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_13.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['recall'])
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
start = time.time()
history = model_13.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=16, epochs=100, #smaller batch sizes
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn( /usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 53ms/step - loss: 0.8289 - recall: 0.9091
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 37s 57ms/step - loss: 0.8287 - recall: 0.9092 - val_loss: 1.0147 - val_recall: 1.0000 Epoch 2/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 37s 58ms/step - loss: 0.6650 - recall: 0.9884 - val_loss: 0.9578 - val_recall: 0.9540 Epoch 3/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.6284 - recall: 0.9711 - val_loss: 0.9190 - val_recall: 0.9417 Epoch 4/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.6048 - recall: 0.9571 - val_loss: 0.9286 - val_recall: 0.9509 Epoch 5/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.5780 - recall: 0.9577 - val_loss: 0.9301 - val_recall: 0.9540 Epoch 6/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.5650 - recall: 0.9557 - val_loss: 0.9820 - val_recall: 0.9632 Epoch 7/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 37s 58ms/step - loss: 0.5467 - recall: 0.9501 - val_loss: 1.0206 - val_recall: 0.9632 Epoch 8/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 36s 57ms/step - loss: 0.5400 - recall: 0.9520 - val_loss: 0.9838 - val_recall: 0.9632 Time taken in seconds 291.93304467201233
plot(history,'loss')
plot(history,'recall')
val_pred = model_13.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
28/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step precision recall f1-score support 0.0 0.95 0.31 0.47 1274 1.0 0.26 0.94 0.41 326 accuracy 0.44 1600 macro avg 0.61 0.63 0.44 1600 weighted avg 0.81 0.44 0.45 1600
results.loc[13] = [3, "128,64,32", "leakyrelu,leakyrelu,relu", 100, 16, "SGD", "0.001, -", "xavier", "L2,N/A", history.history["loss"][-1], history.history["val_loss"][-1], history.history["recall"][-1], history.history["val_recall"][-1], round(end-start,2)]
results
| # hidden layers | # neurons - hidden layer | activation function - hidden layer | # epochs | batch size | optimizer | learning rate | initializer | regularizer | train loss | validation loss | train recall | validation recall | time (secs) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | 128,64 | relu,relu | 10 | 16 | sgd | 0.001, - | xavier | - | 0.459178 | 0.455843 | 0.000000 | 0.003067 | 143.87 |
| 1 | 2 | 64,32 | relu,relu | 50 | 16 | Adam | 0, - | xavier | - | 0.261802 | 0.385848 | 0.608129 | 0.457055 | 606.90 |
| 3 | 2 | 128,64 | relu,relu | 100 | 32 | SGD | 0.001, - | xavier | - | 0.389204 | 0.717597 | 0.816130 | 0.858896 | 332.36 |
| 4 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.574763 | 0.659100 | 0.776295 | 0.797546 | 264.18 |
| 5 | 2 | 128,64 | relu,relu | 100 | 32 | Adam | 0, - | xavier | L2,L1 | 0.640411 | 0.693025 | 0.772174 | 0.806748 | 198.04 |
| 6 | 2 | 128,64 | relu,relu | 50 | 8 | Adam | 0, - | xavier | L2,L1 | 0.550212 | 0.607145 | 0.777473 | 0.779141 | 681.80 |
| 7 | 2 | 64,32 | tanh,tanh | 50 | 32 | SGD | 0.001, - | xavier | - | 0.417627 | 0.492607 | 0.799843 | 0.736196 | 404.54 |
| 8 | 2 | 32,16 | sigmoid,sigmoid | 50 | 32 | SGD | 0.0001, - | xavier | - | 0.680181 | 0.689308 | 0.696625 | 0.723926 | 490.79 |
| 9 | 2 | 128,64 | relu,relu | 150 | 32 | SGD | 0.001, - | xavier | - | 0.434032 | 0.556475 | 0.788265 | 0.806748 | 135.95 |
| 10 | 2 | 128,64 | relu,relu | 100 | 16 | SGD | 0.001, - | xavier | - | 0.598579 | 0.846530 | 0.994309 | 0.957055 | 2449.96 |
| 11 | 2 | 128,64 | relu,relu | 50 | 16 | SGD | 0.001, - | xavier | L2,N/A | 0.575172 | 1.231755 | 0.947214 | 0.969325 | 922.24 |
| 12 | 3 | 128,64,32 | leakyrelu,leakyrelu,relu | 100 | 32 | SGD | 0.001, - | xavier | L2,L2,N/A | 0.643929 | 0.973020 | 0.965463 | 0.960123 | 151.73 |
| 13 | 3 | 128,64,32 | leakyrelu,leakyrelu,relu | 100 | 16 | SGD | 0.001, - | xavier | L2,N/A | 0.537729 | 0.983835 | 0.952512 | 0.963190 | 291.93 |
more agressive class weight
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Dense, Dropout, LeakyReLU, Input
from tensorflow.keras.models import Sequential
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.utils.class_weight import compute_class_weight
from tensorflow.keras import regularizers
import time
import numpy as np
tf.keras.backend.clear_session()
# Custom F1 score metric for the minority class (target variable)
def f1_minority(y_true, y_pred):
y_true = tf.cast(y_true, tf.float32)
y_pred = tf.cast(y_pred, tf.float32)
tp = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
pp = K.sum(K.round(K.clip(y_pred, 0, 1)))
fp = pp - tp
fn = K.sum(K.round(K.clip(y_true, 0, 1))) - tp
precision = tp / (tp + fp + K.epsilon())
recall = tp / (tp + fn + K.epsilon())
f1_val = 2 * ((precision * recall) / (precision + recall + K.epsilon()))
return f1_val
model_14 = Sequential()
#
model_14.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train_smote.shape[1],
kernel_initializer='glorot_uniform',
kernel_regularizer=regularizers.l2(0.0001)))
model_14.add(Dropout(0.2))
model_14.add(Dense(64, activation=LeakyReLU(alpha=0.1),
kernel_initializer='glorot_uniform',
kernel_regularizer=regularizers.l2(0.0001)))
model_14.add(Dropout(0.2))
model_14.add(Dense(32, activation='relu', kernel_initializer='glorot_uniform'))
model_14.add(Dense(1, activation='sigmoid', kernel_initializer='glorot_uniform'))
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: 1, 1: 1.5} #slightly more aggressive
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
# Compile the model with the custom F1 score metric
model_14.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[f1_minority])
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
start = time.time()
history = model_14.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=16, epochs=100,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn( /usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
636/637 ━━━━━━━━━━━━━━━━━━━━ 0s 49ms/step - f1_minority: 0.6327 - loss: 0.8170
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 52ms/step - f1_minority: 0.6329 - loss: 0.8169 - val_f1_minority: 0.4314 - val_loss: 0.7209 Epoch 2/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 51ms/step - f1_minority: 0.7186 - loss: 0.7075 - val_f1_minority: 0.4467 - val_loss: 0.6953 Epoch 3/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 52ms/step - f1_minority: 0.7386 - loss: 0.6634 - val_f1_minority: 0.4564 - val_loss: 0.6698 Epoch 4/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 32s 51ms/step - f1_minority: 0.7466 - loss: 0.6492 - val_f1_minority: 0.4524 - val_loss: 0.6592 Epoch 5/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 51ms/step - f1_minority: 0.7749 - loss: 0.5898 - val_f1_minority: 0.4580 - val_loss: 0.6729 Epoch 6/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 51ms/step - f1_minority: 0.7721 - loss: 0.5809 - val_f1_minority: 0.4480 - val_loss: 0.7282 Epoch 7/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 51ms/step - f1_minority: 0.7817 - loss: 0.5772 - val_f1_minority: 0.4491 - val_loss: 0.7107 Epoch 8/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 32s 51ms/step - f1_minority: 0.7859 - loss: 0.5628 - val_f1_minority: 0.4536 - val_loss: 0.7064 Epoch 9/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 33s 52ms/step - f1_minority: 0.7863 - loss: 0.5537 - val_f1_minority: 0.4453 - val_loss: 0.7374 Time taken in seconds 294.83499097824097
plot(history,'loss')
plot(history,'f1_minority')
val_pred = model_14.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
28/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step precision recall f1-score support 0.0 0.94 0.57 0.71 1274 1.0 0.34 0.87 0.49 326 accuracy 0.63 1600 macro avg 0.64 0.72 0.60 1600 weighted avg 0.82 0.63 0.67 1600
this model is better at not wasting resaources butthere is opportunity cost here from worse performance on predicting churners
higher patience, higher dropout glourot a new adjusted l2 regulizer higher dropout and patience
model_15 = Sequential()
model_15.add(Dense(128, activation=LeakyReLU(alpha=0.1),
input_dim=X_train_smote.shape[1],
kernel_initializer='glorot_uniform',
kernel_regularizer=regularizers.l2(0.001))) # L2 regularization adjusted
model_15.add(BatchNormalization())
model_15.add(Dropout(0.5))
model_15.add(Dense(64, activation=LeakyReLU(alpha=0.1),
kernel_initializer='glorot_uniform',
kernel_regularizer=regularizers.l2(0.001))) # L2 regularization adjusted
model_15.add(BatchNormalization())
model_15.add(Dropout(0.5))
model_15.add(Dense(32, activation='relu',
kernel_initializer='glorot_uniform',
kernel_regularizer=regularizers.l2(0.001))) # L2 regularization adjusted
model_15.add(BatchNormalization())
model_15.add(Dropout(0.5))
model_15.add(Dense(1, activation='sigmoid'))
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: 1, 1: 1.5} #same as before
# Compile the model with a custom F1 score metric for our minority target variable class
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_15.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[f1_minority])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True) #decent patience
start = time.time()
history = model_15.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=16, epochs=100,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn( /usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 87ms/step - f1_minority: 0.5957 - loss: 1.1143
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.5958 - loss: 1.1141 - val_f1_minority: 0.3806 - val_loss: 0.9081 Epoch 2/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 91ms/step - f1_minority: 0.6854 - loss: 0.8923 - val_f1_minority: 0.4058 - val_loss: 0.8755 Epoch 3/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.6949 - loss: 0.8710 - val_f1_minority: 0.4172 - val_loss: 0.8515 Epoch 4/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7090 - loss: 0.8507 - val_f1_minority: 0.4228 - val_loss: 0.8509 Epoch 5/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7161 - loss: 0.8366 - val_f1_minority: 0.4457 - val_loss: 0.8071 Epoch 6/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7279 - loss: 0.8226 - val_f1_minority: 0.4230 - val_loss: 0.8460 Epoch 7/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.7250 - loss: 0.8167 - val_f1_minority: 0.4283 - val_loss: 0.8336 Epoch 8/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 91ms/step - f1_minority: 0.7430 - loss: 0.7925 - val_f1_minority: 0.4372 - val_loss: 0.8042 Epoch 9/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 91ms/step - f1_minority: 0.7363 - loss: 0.7887 - val_f1_minority: 0.4381 - val_loss: 0.8132 Epoch 10/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7271 - loss: 0.7869 - val_f1_minority: 0.4350 - val_loss: 0.8053 Epoch 11/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7407 - loss: 0.7692 - val_f1_minority: 0.4550 - val_loss: 0.7628 Epoch 12/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7399 - loss: 0.7720 - val_f1_minority: 0.4346 - val_loss: 0.8022 Epoch 13/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.7510 - loss: 0.7539 - val_f1_minority: 0.4424 - val_loss: 0.7799 Epoch 14/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.7518 - loss: 0.7638 - val_f1_minority: 0.4452 - val_loss: 0.7882 Epoch 15/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7464 - loss: 0.7516 - val_f1_minority: 0.4475 - val_loss: 0.7753 Epoch 16/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 90ms/step - f1_minority: 0.7691 - loss: 0.7205 - val_f1_minority: 0.4542 - val_loss: 0.7518 Epoch 17/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7569 - loss: 0.7293 - val_f1_minority: 0.4534 - val_loss: 0.7517 Epoch 18/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 58s 91ms/step - f1_minority: 0.7577 - loss: 0.7110 - val_f1_minority: 0.4440 - val_loss: 0.7738 Epoch 19/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7552 - loss: 0.7164 - val_f1_minority: 0.4492 - val_loss: 0.7561 Epoch 20/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7520 - loss: 0.7141 - val_f1_minority: 0.4463 - val_loss: 0.7544 Epoch 21/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7585 - loss: 0.7130 - val_f1_minority: 0.4426 - val_loss: 0.7661 Epoch 22/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7545 - loss: 0.7057 - val_f1_minority: 0.4367 - val_loss: 0.7783 Epoch 23/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7554 - loss: 0.7099 - val_f1_minority: 0.4440 - val_loss: 0.7754 Epoch 24/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7635 - loss: 0.6940 - val_f1_minority: 0.4473 - val_loss: 0.7620 Epoch 25/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7573 - loss: 0.6948 - val_f1_minority: 0.4533 - val_loss: 0.7241 Epoch 26/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7634 - loss: 0.6957 - val_f1_minority: 0.4411 - val_loss: 0.7467 Epoch 27/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7659 - loss: 0.6715 - val_f1_minority: 0.4508 - val_loss: 0.7379 Epoch 28/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7610 - loss: 0.6921 - val_f1_minority: 0.4461 - val_loss: 0.7604 Epoch 29/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7753 - loss: 0.6693 - val_f1_minority: 0.4529 - val_loss: 0.7251 Epoch 30/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7637 - loss: 0.6756 - val_f1_minority: 0.4506 - val_loss: 0.7562 Epoch 31/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7685 - loss: 0.6712 - val_f1_minority: 0.4508 - val_loss: 0.7489 Epoch 32/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7679 - loss: 0.6755 - val_f1_minority: 0.4472 - val_loss: 0.7366 Epoch 33/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7799 - loss: 0.6616 - val_f1_minority: 0.4478 - val_loss: 0.7465 Epoch 34/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 89ms/step - f1_minority: 0.7713 - loss: 0.6557 - val_f1_minority: 0.4409 - val_loss: 0.7512 Epoch 35/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7701 - loss: 0.6673 - val_f1_minority: 0.4498 - val_loss: 0.7086 Epoch 36/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 89ms/step - f1_minority: 0.7689 - loss: 0.6642 - val_f1_minority: 0.4460 - val_loss: 0.7305 Epoch 37/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7683 - loss: 0.6573 - val_f1_minority: 0.4492 - val_loss: 0.7272 Epoch 38/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 89ms/step - f1_minority: 0.7640 - loss: 0.6598 - val_f1_minority: 0.4526 - val_loss: 0.7316 Epoch 39/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7686 - loss: 0.6618 - val_f1_minority: 0.4497 - val_loss: 0.7461 Epoch 40/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7680 - loss: 0.6513 - val_f1_minority: 0.4505 - val_loss: 0.7243 Epoch 41/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7773 - loss: 0.6408 - val_f1_minority: 0.4420 - val_loss: 0.7539 Epoch 42/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 88ms/step - f1_minority: 0.7636 - loss: 0.6563 - val_f1_minority: 0.4438 - val_loss: 0.7698 Epoch 43/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7807 - loss: 0.6462 - val_f1_minority: 0.4409 - val_loss: 0.7685 Epoch 44/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7702 - loss: 0.6436 - val_f1_minority: 0.4463 - val_loss: 0.7264 Epoch 45/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7667 - loss: 0.6388 - val_f1_minority: 0.4508 - val_loss: 0.7214 Time taken in seconds 2564.5972447395325
plot(history,'loss')
plot(history,'f1_minority')
val_pred = model_15.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/50 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step precision recall f1-score support 0.0 0.95 0.55 0.70 1274 1.0 0.33 0.88 0.48 326 accuracy 0.62 1600 macro avg 0.64 0.71 0.59 1600 weighted avg 0.82 0.62 0.65 1600
not bad but i want to try more agressive class weights
more regulizer layers lower patience lower learning rate higher momentum and even more agressive class weights
import warnings
warnings.filterwarnings("ignore", category=UserWarning, module="keras")
model_16 = Sequential()
model_16.add(Input(shape=(X_train_smote.shape[1],)))
model_16.add(Dense(128, activation=LeakyReLU(negative_slope=0.1), # Use negative_slope
kernel_initializer='glorot_uniform',
kernel_regularizer=regularizers.l2(0.001)))
model_16.add(BatchNormalization())
model_16.add(Dropout(0.5))
model_16.add(Dense(64, activation=LeakyReLU(negative_slope=0.1), # Use negative_slope
kernel_initializer='glorot_uniform',
kernel_regularizer=regularizers.l2(0.001)))
model_16.add(BatchNormalization())
model_16.add(Dropout(0.5))
model_16.add(Dense(32, activation='relu',
kernel_initializer='glorot_uniform',
kernel_regularizer=regularizers.l2(0.001)))
model_16.add(BatchNormalization())
model_16.add(Dropout(0.5))
model_16.add(Dense(1, activation='sigmoid'))
#f1_minority
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_16.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[f1_minority])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
start = time.time()
history = model_16.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=16, epochs=100,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 86ms/step - f1_minority: 0.5806 - loss: 1.1193
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.5807 - loss: 1.1191 - val_f1_minority: 0.3761 - val_loss: 0.9355 Epoch 2/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.6936 - loss: 0.8861 - val_f1_minority: 0.4011 - val_loss: 0.8910 Epoch 3/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7009 - loss: 0.8579 - val_f1_minority: 0.4014 - val_loss: 0.8779 Epoch 4/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7145 - loss: 0.8463 - val_f1_minority: 0.4115 - val_loss: 0.8730 Epoch 5/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7319 - loss: 0.8179 - val_f1_minority: 0.4236 - val_loss: 0.8467 Epoch 6/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 56s 89ms/step - f1_minority: 0.7291 - loss: 0.8179 - val_f1_minority: 0.4161 - val_loss: 0.8509 Epoch 7/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7318 - loss: 0.8117 - val_f1_minority: 0.4323 - val_loss: 0.8470 Epoch 8/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7337 - loss: 0.7934 - val_f1_minority: 0.4279 - val_loss: 0.8326 Epoch 9/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7480 - loss: 0.7689 - val_f1_minority: 0.4365 - val_loss: 0.8110 Epoch 10/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7408 - loss: 0.7812 - val_f1_minority: 0.4284 - val_loss: 0.8193 Epoch 11/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7454 - loss: 0.7751 - val_f1_minority: 0.4319 - val_loss: 0.8110 Epoch 12/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7529 - loss: 0.7551 - val_f1_minority: 0.4395 - val_loss: 0.8000 Epoch 13/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7430 - loss: 0.7604 - val_f1_minority: 0.4522 - val_loss: 0.7826 Epoch 14/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7571 - loss: 0.7442 - val_f1_minority: 0.4455 - val_loss: 0.7911 Epoch 15/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 90ms/step - f1_minority: 0.7468 - loss: 0.7522 - val_f1_minority: 0.4499 - val_loss: 0.7700 Epoch 16/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7583 - loss: 0.7192 - val_f1_minority: 0.4379 - val_loss: 0.7976 Epoch 17/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7646 - loss: 0.7195 - val_f1_minority: 0.4387 - val_loss: 0.7910 Epoch 18/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7576 - loss: 0.7252 - val_f1_minority: 0.4435 - val_loss: 0.7734 Epoch 19/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7662 - loss: 0.7018 - val_f1_minority: 0.4429 - val_loss: 0.7511 Epoch 20/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7627 - loss: 0.7083 - val_f1_minority: 0.4380 - val_loss: 0.7873 Epoch 21/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7564 - loss: 0.7084 - val_f1_minority: 0.4403 - val_loss: 0.7926 Epoch 22/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7601 - loss: 0.7032 - val_f1_minority: 0.4445 - val_loss: 0.7622 Epoch 23/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7571 - loss: 0.7087 - val_f1_minority: 0.4451 - val_loss: 0.7670 Epoch 24/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7637 - loss: 0.6925 - val_f1_minority: 0.4396 - val_loss: 0.7681 Epoch 25/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7710 - loss: 0.6928 - val_f1_minority: 0.4419 - val_loss: 0.7659 Epoch 26/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7711 - loss: 0.6761 - val_f1_minority: 0.4420 - val_loss: 0.7681 Epoch 27/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7610 - loss: 0.6906 - val_f1_minority: 0.4404 - val_loss: 0.7866 Epoch 28/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7592 - loss: 0.6856 - val_f1_minority: 0.4386 - val_loss: 0.7874 Epoch 29/100 637/637 ━━━━━━━━━━━━━━━━━━━━ 57s 89ms/step - f1_minority: 0.7651 - loss: 0.6864 - val_f1_minority: 0.4435 - val_loss: 0.7734 Time taken in seconds 1647.1443145275116
plot(history,'loss')
plot(history,'f1_minority')
val_pred = model_16.predict(X_val)
val_pred = (val_pred > 0.50) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/50 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step precision recall f1-score support 0.0 0.95 0.51 0.67 1274 1.0 0.32 0.90 0.47 326 accuracy 0.59 1600 macro avg 0.64 0.71 0.57 1600 weighted avg 0.83 0.59 0.63 1600
df.shape
tf.keras.backend.clear_session()
model_17 = Sequential()
model_17.add(Dense(64, kernel_regularizer=regularizers.l2(0.001),
kernel_initializer='glorot_uniform', input_dim=X_train.shape[1])) #not using smote or scaled
model_17.add(LeakyReLU(alpha=0.1))
model_17.add(BatchNormalization())
model_17.add(Dropout(0.4))
model_17.add(Dense(32, kernel_regularizer=regularizers.l2(0.001),
kernel_initializer='glorot_uniform'))
model_17.add(LeakyReLU(alpha=0.1))
model_17.add(BatchNormalization())
model_17.add(Dropout(0.4))
model_17.add(Dense(1, activation='sigmoid'))
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: 1, 1: 2}
optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.99)
model_17.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy']) #accuracy but with aggressive class weights and no smote
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
start = time.time()
history = model_17.fit(X_train, y_train,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=8, epochs=25, # Adjusted batch size
callbacks=[early_stopping])
Epoch 1/25
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
800/800 ━━━━━━━━━━━━━━━━━━━━ 0s 64ms/step - accuracy: 0.6619 - loss: 1.5933
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6619 - loss: 1.5936 - val_accuracy: 0.7875 - val_loss: 1.1769 Epoch 2/25 800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6587 - loss: 2.0130 - val_accuracy: 0.7025 - val_loss: 1.0165 Epoch 3/25 800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6745 - loss: 1.4768 - val_accuracy: 0.7856 - val_loss: 0.9387 Epoch 4/25 800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6611 - loss: 1.4657 - val_accuracy: 0.7600 - val_loss: 1.2144 Epoch 5/25 800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6827 - loss: 1.9323 - val_accuracy: 0.7956 - val_loss: 1.5615 Epoch 6/25 800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 68ms/step - accuracy: 0.6458 - loss: 3.0731 - val_accuracy: 0.7106 - val_loss: 1.7047 Epoch 7/25 800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 68ms/step - accuracy: 0.6573 - loss: 3.6581 - val_accuracy: 0.7744 - val_loss: 5.0377 Epoch 8/25 800/800 ━━━━━━━━━━━━━━━━━━━━ 55s 69ms/step - accuracy: 0.6587 - loss: 9.9355 - val_accuracy: 0.7013 - val_loss: 6.3116
plot(history,'loss')
plot(history,'accuracy')
val_pred = model_17.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
15/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step precision recall f1-score support 0.0 0.80 0.98 0.88 1274 1.0 0.24 0.02 0.04 326 accuracy 0.79 1600 macro avg 0.52 0.50 0.46 1600 weighted avg 0.68 0.79 0.71 1600
smaller regularization larger learning rate smaller momentum and smaller dropout
tf.keras.backend.clear_session()
# Define macro F1-score metric
def macro_f1(y_true, y_pred):
y_pred = K.round(y_pred) # Round predictions to 0 or 1
tp = K.sum(K.cast(y_true * y_pred, 'float'), axis=0)
fp = K.sum(K.cast((1 - y_true) * y_pred, 'float'), axis=0)
fn = K.sum(K.cast(y_true * (1 - y_pred), 'float'), axis=0)
p = tp / (tp + fp + K.epsilon())
r = tp / (tp + fn + K.epsilon())
f1 = 2 * p * r / (p + r + K.epsilon())
macro_f1 = K.mean(f1)
return macro_f1
tf.keras.backend.clear_session()
# Define model_18 (renamed from model_20)
model_18 = Sequential()
# Input layer with L2 regularization and LeakyReLU activation
model_18.add(Dense(128, activation=LeakyReLU(alpha=0.05),
input_dim=X_train_smote.shape[1],
kernel_regularizer=regularizers.l2(0.001)))
model_18.add(BatchNormalization())
model_18.add(Dropout(0.4))
# Hidden layer with LeakyReLU activation and L2 regularization
model_18.add(Dense(64, activation=LeakyReLU(alpha=0.05),
kernel_regularizer=regularizers.l2(0.001)))
model_18.add(BatchNormalization())
model_18.add(Dropout(0.4))
# Hidden layer with ReLU activation
model_18.add(Dense(32, activation='relu'))
model_18.add(Dense(1, activation='sigmoid'))
# Calculate class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]} # Using computed weights
# Compile the model with macro F1-score metric
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001) # Adam optimizer
model_18.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[macro_f1])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
start = time.time()
history = model_18.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=32, epochs=100,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 0s 103ms/step - loss: 0.8649 - macro_f1: 0.6796
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 106ms/step - loss: 0.8647 - macro_f1: 0.6796 - val_loss: 1.0137 - val_macro_f1: 0.4021 Epoch 2/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.7228 - macro_f1: 0.7120 - val_loss: 1.1489 - val_macro_f1: 0.4092 Epoch 3/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.6662 - macro_f1: 0.7414 - val_loss: 1.0782 - val_macro_f1: 0.4267 Epoch 4/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.6430 - macro_f1: 0.7505 - val_loss: 1.0401 - val_macro_f1: 0.4287 Epoch 5/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.6253 - macro_f1: 0.7543 - val_loss: 1.0120 - val_macro_f1: 0.4241 Epoch 6/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5975 - macro_f1: 0.7596 - val_loss: 1.0781 - val_macro_f1: 0.4315 Epoch 7/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5967 - macro_f1: 0.7664 - val_loss: 1.0448 - val_macro_f1: 0.4273 Epoch 8/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5736 - macro_f1: 0.7653 - val_loss: 1.1825 - val_macro_f1: 0.4193 Epoch 9/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5745 - macro_f1: 0.7631 - val_loss: 1.1427 - val_macro_f1: 0.4212 Epoch 10/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5576 - macro_f1: 0.7760 - val_loss: 1.1048 - val_macro_f1: 0.4261 Epoch 11/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 108ms/step - loss: 0.5701 - macro_f1: 0.7656 - val_loss: 1.0891 - val_macro_f1: 0.4318 Epoch 12/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 35s 109ms/step - loss: 0.5521 - macro_f1: 0.7775 - val_loss: 1.0942 - val_macro_f1: 0.4297 Epoch 13/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5439 - macro_f1: 0.7803 - val_loss: 1.1071 - val_macro_f1: 0.4302 Epoch 14/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5460 - macro_f1: 0.7743 - val_loss: 1.0465 - val_macro_f1: 0.4372 Epoch 15/100 319/319 ━━━━━━━━━━━━━━━━━━━━ 34s 107ms/step - loss: 0.5339 - macro_f1: 0.7845 - val_loss: 1.1456 - val_macro_f1: 0.4226 Time taken in seconds 512.9856388568878
plot(history,'loss')
plot(history,'macro_f1')
val_pred = model_18.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
13/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step precision recall f1-score support 0.0 0.97 0.39 0.56 1274 1.0 0.29 0.95 0.44 326 accuracy 0.51 1600 macro avg 0.63 0.67 0.50 1600 weighted avg 0.83 0.51 0.53 1600
we need to use our best moddel to map accuracy using smote and our best parameters and hyperparameters
tf.keras.backend.clear_session()
model_19 = Sequential()
model_19.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train_smote.shape[1],
kernel_regularizer=regularizers.l2(0.0001)))
model_19.add(Dropout(0.2))
model_19.add(Dense(64, activation=LeakyReLU(alpha=0.1)))
model_19.add(Dropout(0.2))
model_19.add(Dense(32, activation='relu'))
model_19.add(Dense(1, activation='sigmoid'))
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}
optimizer = tf.keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
model_19.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_accuracy', patience=5, restore_best_weights=True)
start = time.time()
history = model_19.fit(X_train_smote_scaled, y_train_smote,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=16, epochs=50,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn( /usr/local/lib/python3.11/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 0s 45ms/step - accuracy: 0.4996 - loss: 0.8184
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.4996 - loss: 0.8183 - val_accuracy: 0.2556 - val_loss: 1.0267 Epoch 2/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.5576 - loss: 0.6607 - val_accuracy: 0.3581 - val_loss: 0.9696 Epoch 3/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.6182 - loss: 0.6301 - val_accuracy: 0.3994 - val_loss: 0.9642 Epoch 4/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.6277 - loss: 0.6154 - val_accuracy: 0.4081 - val_loss: 1.0050 Epoch 5/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.6480 - loss: 0.5901 - val_accuracy: 0.4594 - val_loss: 0.9826 Epoch 6/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 31s 48ms/step - accuracy: 0.6771 - loss: 0.5677 - val_accuracy: 0.4712 - val_loss: 0.9927 Epoch 7/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7015 - loss: 0.5447 - val_accuracy: 0.4956 - val_loss: 0.9915 Epoch 8/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7130 - loss: 0.5358 - val_accuracy: 0.4913 - val_loss: 1.0247 Epoch 9/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7084 - loss: 0.5357 - val_accuracy: 0.4994 - val_loss: 1.0145 Epoch 10/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7149 - loss: 0.5233 - val_accuracy: 0.5063 - val_loss: 0.9989 Epoch 11/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7282 - loss: 0.5147 - val_accuracy: 0.4806 - val_loss: 1.0806 Epoch 12/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7188 - loss: 0.5170 - val_accuracy: 0.5006 - val_loss: 1.0140 Epoch 13/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7352 - loss: 0.5021 - val_accuracy: 0.5069 - val_loss: 1.0302 Epoch 14/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 31s 48ms/step - accuracy: 0.7315 - loss: 0.5003 - val_accuracy: 0.4888 - val_loss: 1.1102 Epoch 15/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7294 - loss: 0.5086 - val_accuracy: 0.5250 - val_loss: 0.9752 Epoch 16/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 31s 48ms/step - accuracy: 0.7360 - loss: 0.5005 - val_accuracy: 0.5050 - val_loss: 1.0546 Epoch 17/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7278 - loss: 0.5042 - val_accuracy: 0.5150 - val_loss: 1.0319 Epoch 18/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7393 - loss: 0.4947 - val_accuracy: 0.5150 - val_loss: 0.9952 Epoch 19/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 47ms/step - accuracy: 0.7459 - loss: 0.4815 - val_accuracy: 0.5113 - val_loss: 1.0089 Epoch 20/50 637/637 ━━━━━━━━━━━━━━━━━━━━ 30s 48ms/step - accuracy: 0.7356 - loss: 0.4971 - val_accuracy: 0.5088 - val_loss: 1.0783 Time taken in seconds 606.9027745723724
plot(history,'loss')
plot(history,'accuracy')
val_pred = model_19.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
28/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step precision recall f1-score support 0.0 0.96 0.42 0.58 1274 1.0 0.29 0.94 0.45 326 accuracy 0.53 1600 macro avg 0.63 0.68 0.52 1600 weighted avg 0.83 0.53 0.56 1600
tf.keras.backend.clear_session()
model_20 = Sequential()
model_20.add(Dense(64, activation="tanh", input_dim=X_train.shape[1])) #tanh
model_20.add(Dense(32, activation="tanh"))#tanh
model_20.add(Dense(1, activation='sigmoid'))
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model_20.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
start = time.time()
history = model_20.fit(X_train, y_train,
validation_data=(X_val, y_val),
batch_size=32, epochs=50,
callbacks=[early_stopping])
end = time.time()
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
199/200 ━━━━━━━━━━━━━━━━━━━━ 0s 46ms/step - accuracy: 0.7803 - loss: 0.4802
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.7805 - loss: 0.4798 - val_accuracy: 0.8138 - val_loss: 0.4177 Epoch 2/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.8251 - loss: 0.4018 - val_accuracy: 0.8281 - val_loss: 0.3841 Epoch 3/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.8383 - loss: 0.3783 - val_accuracy: 0.8413 - val_loss: 0.3632 Epoch 4/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.8515 - loss: 0.3559 - val_accuracy: 0.8469 - val_loss: 0.3548 Epoch 5/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8621 - loss: 0.3443 - val_accuracy: 0.8506 - val_loss: 0.3505 Epoch 6/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8638 - loss: 0.3485 - val_accuracy: 0.8519 - val_loss: 0.3480 Epoch 7/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8588 - loss: 0.3418 - val_accuracy: 0.8519 - val_loss: 0.3487 Epoch 8/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 50ms/step - accuracy: 0.8596 - loss: 0.3353 - val_accuracy: 0.8537 - val_loss: 0.3469 Epoch 9/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8647 - loss: 0.3264 - val_accuracy: 0.8519 - val_loss: 0.3463 Epoch 10/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 51ms/step - accuracy: 0.8608 - loss: 0.3353 - val_accuracy: 0.8512 - val_loss: 0.3510 Epoch 11/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 52ms/step - accuracy: 0.8606 - loss: 0.3336 - val_accuracy: 0.8500 - val_loss: 0.3541 Epoch 12/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 49ms/step - accuracy: 0.8635 - loss: 0.3378 - val_accuracy: 0.8481 - val_loss: 0.3467 Epoch 13/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 10s 51ms/step - accuracy: 0.8666 - loss: 0.3351 - val_accuracy: 0.8512 - val_loss: 0.3491 Epoch 14/50 200/200 ━━━━━━━━━━━━━━━━━━━━ 11s 53ms/step - accuracy: 0.8698 - loss: 0.3239 - val_accuracy: 0.8587 - val_loss: 0.3485
plot(history,'loss')
plot(history,'accuracy')
val_pred = model_20.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
25/50 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step precision recall f1-score support 0.0 0.87 0.96 0.91 1274 1.0 0.73 0.44 0.55 326 accuracy 0.85 1600 macro avg 0.80 0.70 0.73 1600 weighted avg 0.84 0.85 0.84 1600
not using the x_train_smote/scaled variables were the problem but we need to keep going because the recall is bad but now the accuracy is good
tf.keras.backend.clear_session()
model_21 = Sequential()
model_21.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train.shape[1],
kernel_regularizer=regularizers.l2(0.0001)))
model_21.add(Dropout(0.2))
model_21.add(Dense(64, activation=LeakyReLU(alpha=0.1)))
model_21.add(Dropout(0.2))
model_21.add(Dense(32, activation='relu'))
model_21.add(Dense(1, activation='sigmoid'))
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}
optimizer = tf.keras.optimizers.Adam()
model_21.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_accuracy', patience=7, restore_best_weights=True)
start = time.time()
history = model_21.fit(X_train, y_train,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=16, epochs=50,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 0s 72ms/step - accuracy: 0.7003 - loss: 0.6147
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 77ms/step - accuracy: 0.7003 - loss: 0.6146 - val_accuracy: 0.7831 - val_loss: 0.4704 Epoch 2/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 76ms/step - accuracy: 0.7577 - loss: 0.4956 - val_accuracy: 0.7594 - val_loss: 0.4859 Epoch 3/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 76ms/step - accuracy: 0.7714 - loss: 0.4891 - val_accuracy: 0.7656 - val_loss: 0.4809 Epoch 4/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 76ms/step - accuracy: 0.7852 - loss: 0.4585 - val_accuracy: 0.7844 - val_loss: 0.4795 Epoch 5/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8118 - loss: 0.4496 - val_accuracy: 0.7819 - val_loss: 0.4761 Epoch 6/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7971 - loss: 0.4517 - val_accuracy: 0.7688 - val_loss: 0.4876 Epoch 7/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7948 - loss: 0.4543 - val_accuracy: 0.7925 - val_loss: 0.4620 Epoch 8/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8072 - loss: 0.4560 - val_accuracy: 0.8050 - val_loss: 0.4427 Epoch 9/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7937 - loss: 0.4708 - val_accuracy: 0.7613 - val_loss: 0.5064 Epoch 10/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7942 - loss: 0.4534 - val_accuracy: 0.7950 - val_loss: 0.4471 Epoch 11/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7937 - loss: 0.4476 - val_accuracy: 0.8100 - val_loss: 0.4328 Epoch 12/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8038 - loss: 0.4510 - val_accuracy: 0.7944 - val_loss: 0.4421 Epoch 13/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8031 - loss: 0.4452 - val_accuracy: 0.8081 - val_loss: 0.4455 Epoch 14/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8047 - loss: 0.4380 - val_accuracy: 0.7875 - val_loss: 0.4603 Epoch 15/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8021 - loss: 0.4450 - val_accuracy: 0.7875 - val_loss: 0.4726 Epoch 16/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8038 - loss: 0.4406 - val_accuracy: 0.7725 - val_loss: 0.4859 Epoch 17/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8014 - loss: 0.4373 - val_accuracy: 0.7875 - val_loss: 0.4707 Epoch 18/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.7966 - loss: 0.4353 - val_accuracy: 0.8169 - val_loss: 0.4269 Epoch 19/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8067 - loss: 0.4344 - val_accuracy: 0.7844 - val_loss: 0.4732 Epoch 20/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8101 - loss: 0.4270 - val_accuracy: 0.7638 - val_loss: 0.4789 Epoch 21/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8172 - loss: 0.4187 - val_accuracy: 0.7900 - val_loss: 0.4585 Epoch 22/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8053 - loss: 0.4317 - val_accuracy: 0.7944 - val_loss: 0.4613 Epoch 23/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8021 - loss: 0.4286 - val_accuracy: 0.8050 - val_loss: 0.4290 Epoch 24/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8042 - loss: 0.4289 - val_accuracy: 0.7875 - val_loss: 0.4645 Epoch 25/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 74ms/step - accuracy: 0.8014 - loss: 0.4323 - val_accuracy: 0.7513 - val_loss: 0.5132 Time taken in seconds 747.0243382453918
plot(history,'loss')
plot(history,'accuracy')
good but we need a lower learning rate more epoch and potentailly some more tuning that we already refined on the other x's and y's
val_pred = model_21.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
19/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step precision recall f1-score support 0.0 0.92 0.84 0.88 1274 1.0 0.54 0.71 0.61 326 accuracy 0.82 1600 macro avg 0.73 0.78 0.75 1600 weighted avg 0.84 0.82 0.83 1600
these results are good just too much oscilation
tf.keras.backend.clear_session()
# Define model_18 (renamed from model_20)
model_22 = Sequential()
# Input layer with L2 regularization and LeakyReLU activation
model_22.add(Dense(128, activation=LeakyReLU(alpha=0.05),
input_dim=X_train.shape[1],
kernel_regularizer=regularizers.l2(0.001)))
model_22.add(BatchNormalization())
model_22.add(Dropout(0.4))
# Hidden layer with LeakyReLU activation and L2 regularization
model_22.add(Dense(64, activation=LeakyReLU(alpha=0.05),
kernel_regularizer=regularizers.l2(0.001)))
model_22.add(BatchNormalization())
model_22.add(Dropout(0.4))
# Hidden layer with ReLU activation
model_22.add(Dense(32, activation='relu'))
model_22.add(Dense(1, activation='sigmoid'))
# Calculate class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]} # Using computed weights
# Compile the model with macro F1-score metric
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001) # Adam optimizer
model_22.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[macro_f1])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
start = time.time()
history = model_22.fit(X_train, y_train,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=32, epochs=100,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 105ms/step - loss: 0.8218 - macro_f1: 0.3913
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.8214 - macro_f1: 0.3915 - val_loss: 0.6059 - val_macro_f1: 0.5133 Epoch 2/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.6514 - macro_f1: 0.4964 - val_loss: 0.5332 - val_macro_f1: 0.5527 Epoch 3/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.6473 - macro_f1: 0.5358 - val_loss: 0.5341 - val_macro_f1: 0.5680 Epoch 4/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 112ms/step - loss: 0.6084 - macro_f1: 0.5207 - val_loss: 0.5449 - val_macro_f1: 0.5680 Epoch 5/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.6105 - macro_f1: 0.5427 - val_loss: 0.5374 - val_macro_f1: 0.5786 Epoch 6/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5971 - macro_f1: 0.5425 - val_loss: 0.5336 - val_macro_f1: 0.5543 Epoch 7/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5860 - macro_f1: 0.5592 - val_loss: 0.5114 - val_macro_f1: 0.5732 Epoch 8/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5721 - macro_f1: 0.5617 - val_loss: 0.5107 - val_macro_f1: 0.5749 Epoch 9/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5725 - macro_f1: 0.5583 - val_loss: 0.5290 - val_macro_f1: 0.5662 Epoch 10/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5242 - macro_f1: 0.5773 - val_loss: 0.5123 - val_macro_f1: 0.5714 Epoch 11/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5478 - macro_f1: 0.5695 - val_loss: 0.5117 - val_macro_f1: 0.5737 Epoch 12/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5396 - macro_f1: 0.5884 - val_loss: 0.5052 - val_macro_f1: 0.5868 Epoch 13/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 112ms/step - loss: 0.5155 - macro_f1: 0.5862 - val_loss: 0.4876 - val_macro_f1: 0.5849 Epoch 14/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 109ms/step - loss: 0.5100 - macro_f1: 0.5678 - val_loss: 0.4919 - val_macro_f1: 0.5911 Epoch 15/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5194 - macro_f1: 0.5736 - val_loss: 0.4913 - val_macro_f1: 0.5886 Epoch 16/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.5303 - macro_f1: 0.5809 - val_loss: 0.4704 - val_macro_f1: 0.5874 Epoch 17/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5052 - macro_f1: 0.6090 - val_loss: 0.4920 - val_macro_f1: 0.5931 Epoch 18/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 23s 112ms/step - loss: 0.5138 - macro_f1: 0.5832 - val_loss: 0.4952 - val_macro_f1: 0.5728 Epoch 19/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 112ms/step - loss: 0.5190 - macro_f1: 0.5798 - val_loss: 0.4867 - val_macro_f1: 0.5726 Epoch 20/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.4949 - macro_f1: 0.5737 - val_loss: 0.4892 - val_macro_f1: 0.5585 Epoch 21/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 111ms/step - loss: 0.5203 - macro_f1: 0.5758 - val_loss: 0.4801 - val_macro_f1: 0.5798 Epoch 22/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 109ms/step - loss: 0.5031 - macro_f1: 0.5730 - val_loss: 0.4957 - val_macro_f1: 0.5860 Epoch 23/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 109ms/step - loss: 0.5026 - macro_f1: 0.5720 - val_loss: 0.4879 - val_macro_f1: 0.5823 Epoch 24/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.4985 - macro_f1: 0.5729 - val_loss: 0.4967 - val_macro_f1: 0.5763 Epoch 25/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 109ms/step - loss: 0.4826 - macro_f1: 0.5834 - val_loss: 0.4803 - val_macro_f1: 0.5772 Epoch 26/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 22s 110ms/step - loss: 0.4904 - macro_f1: 0.5628 - val_loss: 0.4847 - val_macro_f1: 0.5839 Time taken in seconds 575.4505610466003
plot(history,'loss')
plot(history,'macro_f1')
val_pred = model_22.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
13/50 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step precision recall f1-score support 0.0 0.92 0.82 0.87 1274 1.0 0.52 0.74 0.61 326 accuracy 0.81 1600 macro avg 0.72 0.78 0.74 1600 weighted avg 0.84 0.81 0.82 1600
slightly worse performance but fewer oscilations
tf.keras.backend.clear_session()
# Define model_18 (renamed from model_20)
model_23 = Sequential()
# Input layer with L2 regularization and LeakyReLU activation
model_23.add(Dense(64, activation=LeakyReLU(alpha=0.05),
input_dim=X_train.shape[1],
kernel_regularizer=regularizers.l2(0.001)))
model_23.add(BatchNormalization())
model_23.add(Dropout(0.2))
# Hidden layer with LeakyReLU activation and L2 regularization
model_23.add(Dense(32, activation=LeakyReLU(alpha=0.05),
kernel_regularizer=regularizers.l2(0.001)))
model_23.add(BatchNormalization())
model_23.add(Dropout(0.2))
model_23.add(Dense(16, activation=LeakyReLU(alpha=0.05),
kernel_regularizer=regularizers.l2(0.001)))
model_23.add(BatchNormalization())
model_23.add(Dropout(0.2))
model_23.add(Dense(1, activation='sigmoid'))
# Calculate class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]} # Using computed weights
# Compile the model with macro F1-score metric
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001) # Adam optimizer
model_23.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001) # Adam optimizer
model_23.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
start = time.time()
history = model_23.fit(X_train, y_train,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=32, epochs=100,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/100
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
200/200 ━━━━━━━━━━━━━━━━━━━━ 0s 127ms/step - accuracy: 0.5666 - loss: 0.9134
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.5668 - loss: 0.9128 - val_accuracy: 0.7819 - val_loss: 0.6017 Epoch 2/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 136ms/step - accuracy: 0.6775 - loss: 0.7094 - val_accuracy: 0.7875 - val_loss: 0.5680 Epoch 3/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 136ms/step - accuracy: 0.7156 - loss: 0.6501 - val_accuracy: 0.7969 - val_loss: 0.5429 Epoch 4/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7293 - loss: 0.6135 - val_accuracy: 0.7788 - val_loss: 0.5509 Epoch 5/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7317 - loss: 0.6280 - val_accuracy: 0.7931 - val_loss: 0.5280 Epoch 6/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 136ms/step - accuracy: 0.7446 - loss: 0.6027 - val_accuracy: 0.7994 - val_loss: 0.5195 Epoch 7/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7442 - loss: 0.5820 - val_accuracy: 0.8019 - val_loss: 0.5174 Epoch 8/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7538 - loss: 0.5780 - val_accuracy: 0.8012 - val_loss: 0.5144 Epoch 9/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7599 - loss: 0.5753 - val_accuracy: 0.7788 - val_loss: 0.5293 Epoch 10/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7622 - loss: 0.5727 - val_accuracy: 0.7981 - val_loss: 0.5164 Epoch 11/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7718 - loss: 0.5711 - val_accuracy: 0.7862 - val_loss: 0.5230 Epoch 12/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 130ms/step - accuracy: 0.7707 - loss: 0.5395 - val_accuracy: 0.7831 - val_loss: 0.5213 Epoch 13/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7730 - loss: 0.5358 - val_accuracy: 0.8025 - val_loss: 0.4969 Epoch 14/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7709 - loss: 0.5439 - val_accuracy: 0.7844 - val_loss: 0.5134 Epoch 15/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 135ms/step - accuracy: 0.7662 - loss: 0.5376 - val_accuracy: 0.7994 - val_loss: 0.5040 Epoch 16/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7731 - loss: 0.5368 - val_accuracy: 0.7956 - val_loss: 0.4972 Epoch 17/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7688 - loss: 0.5256 - val_accuracy: 0.7906 - val_loss: 0.4961 Epoch 18/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7851 - loss: 0.5212 - val_accuracy: 0.7950 - val_loss: 0.5039 Epoch 19/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7926 - loss: 0.5000 - val_accuracy: 0.7944 - val_loss: 0.4968 Epoch 20/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7681 - loss: 0.5320 - val_accuracy: 0.8006 - val_loss: 0.4747 Epoch 21/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7806 - loss: 0.5133 - val_accuracy: 0.7850 - val_loss: 0.4904 Epoch 22/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7723 - loss: 0.5221 - val_accuracy: 0.7881 - val_loss: 0.5008 Epoch 23/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7717 - loss: 0.5285 - val_accuracy: 0.7825 - val_loss: 0.4961 Epoch 24/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7712 - loss: 0.5201 - val_accuracy: 0.7825 - val_loss: 0.4991 Epoch 25/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7853 - loss: 0.5223 - val_accuracy: 0.8069 - val_loss: 0.4723 Epoch 26/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7894 - loss: 0.4974 - val_accuracy: 0.8062 - val_loss: 0.4768 Epoch 27/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7753 - loss: 0.5096 - val_accuracy: 0.8000 - val_loss: 0.4773 Epoch 28/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7800 - loss: 0.5011 - val_accuracy: 0.7719 - val_loss: 0.4967 Epoch 29/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7707 - loss: 0.5067 - val_accuracy: 0.8094 - val_loss: 0.4715 Epoch 30/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 137ms/step - accuracy: 0.7808 - loss: 0.4989 - val_accuracy: 0.8037 - val_loss: 0.4773 Epoch 31/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7874 - loss: 0.4984 - val_accuracy: 0.7881 - val_loss: 0.4840 Epoch 32/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7877 - loss: 0.4763 - val_accuracy: 0.7962 - val_loss: 0.4717 Epoch 33/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7861 - loss: 0.4852 - val_accuracy: 0.8031 - val_loss: 0.4720 Epoch 34/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7856 - loss: 0.4966 - val_accuracy: 0.8006 - val_loss: 0.4632 Epoch 35/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7837 - loss: 0.4958 - val_accuracy: 0.7944 - val_loss: 0.4737 Epoch 36/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7821 - loss: 0.5083 - val_accuracy: 0.7987 - val_loss: 0.4710 Epoch 37/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7853 - loss: 0.5016 - val_accuracy: 0.7900 - val_loss: 0.4816 Epoch 38/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7845 - loss: 0.4903 - val_accuracy: 0.7962 - val_loss: 0.4696 Epoch 39/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7861 - loss: 0.4850 - val_accuracy: 0.7906 - val_loss: 0.4825 Epoch 40/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 132ms/step - accuracy: 0.7826 - loss: 0.4825 - val_accuracy: 0.8050 - val_loss: 0.4568 Epoch 41/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 132ms/step - accuracy: 0.7794 - loss: 0.4824 - val_accuracy: 0.7825 - val_loss: 0.4873 Epoch 42/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7804 - loss: 0.4924 - val_accuracy: 0.8050 - val_loss: 0.4624 Epoch 43/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 133ms/step - accuracy: 0.7906 - loss: 0.5049 - val_accuracy: 0.7956 - val_loss: 0.4718 Epoch 44/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7919 - loss: 0.4742 - val_accuracy: 0.8012 - val_loss: 0.4710 Epoch 45/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7845 - loss: 0.4845 - val_accuracy: 0.7763 - val_loss: 0.4790 Epoch 46/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7832 - loss: 0.4694 - val_accuracy: 0.8025 - val_loss: 0.4571 Epoch 47/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7937 - loss: 0.4948 - val_accuracy: 0.7819 - val_loss: 0.4800 Epoch 48/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 131ms/step - accuracy: 0.7875 - loss: 0.4751 - val_accuracy: 0.7994 - val_loss: 0.4709 Epoch 49/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 27s 134ms/step - accuracy: 0.7700 - loss: 0.4820 - val_accuracy: 0.7950 - val_loss: 0.4686 Epoch 50/100 200/200 ━━━━━━━━━━━━━━━━━━━━ 26s 132ms/step - accuracy: 0.7881 - loss: 0.4876 - val_accuracy: 0.7919 - val_loss: 0.4678 Time taken in seconds 1332.7293946743011
plot(history,'loss')
plot(history,'accuracy')
val_pred = model_23.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/50 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step precision recall f1-score support 0.0 0.92 0.83 0.87 1274 1.0 0.52 0.72 0.60 326 accuracy 0.81 1600 macro avg 0.72 0.77 0.74 1600 weighted avg 0.84 0.81 0.82 1600
tf.keras.backend.clear_session()
model_24 = Sequential()
model_24.add(Dense(128, activation=LeakyReLU(alpha=0.1), input_dim=X_train.shape[1],
kernel_regularizer=regularizers.l2(0.0001)))
model_24.add(Dropout(0.2))
model_24.add(Dense(64, activation=LeakyReLU(alpha=0.1)))
model_24.add(Dropout(0.2))
model_24.add(Dense(32, activation='relu'))
model_24.add(Dense(1, activation='sigmoid'))
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights_dict = {0: class_weights[0], 1: class_weights[1]}
optimizer = tf.keras.optimizers.Adam()
model_24.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='val_accuracy', patience=7, restore_best_weights=True)
start = time.time()
history = model_24.fit(X_train, y_train,
validation_data=(X_val, y_val),
class_weight=class_weights_dict,
batch_size=16, epochs=50,
callbacks=[early_stopping])
end = time.time()
print("Time taken in seconds ", end - start)
Epoch 1/50
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 0s 71ms/step - accuracy: 0.6154 - loss: 0.6186
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 76ms/step - accuracy: 0.6156 - loss: 0.6185 - val_accuracy: 0.7375 - val_loss: 0.5140 Epoch 2/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 78ms/step - accuracy: 0.7518 - loss: 0.5142 - val_accuracy: 0.7900 - val_loss: 0.4582 Epoch 3/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 77ms/step - accuracy: 0.7776 - loss: 0.4878 - val_accuracy: 0.7481 - val_loss: 0.5360 Epoch 4/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 78ms/step - accuracy: 0.7786 - loss: 0.4882 - val_accuracy: 0.8006 - val_loss: 0.4557 Epoch 5/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 76ms/step - accuracy: 0.7995 - loss: 0.4749 - val_accuracy: 0.7944 - val_loss: 0.4598 Epoch 6/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 76ms/step - accuracy: 0.7996 - loss: 0.4568 - val_accuracy: 0.7975 - val_loss: 0.4542 Epoch 7/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.7945 - loss: 0.4586 - val_accuracy: 0.8256 - val_loss: 0.4129 Epoch 8/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8076 - loss: 0.4520 - val_accuracy: 0.7944 - val_loss: 0.4638 Epoch 9/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 76ms/step - accuracy: 0.8098 - loss: 0.4400 - val_accuracy: 0.8094 - val_loss: 0.4458 Epoch 10/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.7960 - loss: 0.4572 - val_accuracy: 0.8163 - val_loss: 0.4253 Epoch 11/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 77ms/step - accuracy: 0.7972 - loss: 0.4443 - val_accuracy: 0.7925 - val_loss: 0.4552 Epoch 12/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 77ms/step - accuracy: 0.7976 - loss: 0.4464 - val_accuracy: 0.7900 - val_loss: 0.4665 Epoch 13/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 31s 78ms/step - accuracy: 0.8120 - loss: 0.4311 - val_accuracy: 0.8144 - val_loss: 0.4249 Epoch 14/50 400/400 ━━━━━━━━━━━━━━━━━━━━ 30s 75ms/step - accuracy: 0.8093 - loss: 0.4458 - val_accuracy: 0.7962 - val_loss: 0.4533 Time taken in seconds 427.6271617412567
plot(history,'loss')
plot(history,'accuracy')
val_pred = model_24.predict(X_val)
val_pred = (val_pred > 0.5) # Convert probabilities to binary predictions (0 or 1)
# Print the classification report
print(classification_report(y_val, val_pred))
# Create the confusion matrix
cm = confusion_matrix(y_val, val_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
28/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
50/50 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step precision recall f1-score support 0.0 0.91 0.87 0.89 1274 1.0 0.56 0.64 0.60 326 accuracy 0.83 1600 macro avg 0.73 0.76 0.74 1600 weighted avg 0.84 0.83 0.83 1600
despoite our valient efforts we have failed to satisfy risk management and the marklleting department simultaneously with a single model.
Model Performance Comparison and Final Model Selection¶
the scaled smote training varioables messed up a lot of models buy by applying classs weights and using accuracy we were able to build decent models. of these model 23 had the best balance of performance and smoothness
# Generate predictions on the test set
test_pred = model_23.predict(X_test)
test_pred = (test_pred > 0.5) # Convert probabilities to binary predictions
# Print the classification report
print(classification_report(y_test, test_pred))
# Create the confusion matrix
cm = confusion_matrix(y_test, test_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/63 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
63/63 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step precision recall f1-score support 0.0 0.92 0.82 0.86 1593 1.0 0.50 0.71 0.58 407 accuracy 0.79 2000 macro avg 0.71 0.76 0.72 2000 weighted avg 0.83 0.79 0.81 2000
very comprable performance. it may not be what we were hoping for but its not bad. it also flags a significant number of non-churners as potential churners
# Generate predictions on the test set
test_pred = model_23.predict(X_test)
test_pred = (test_pred > 0.3) # Convert probabilities to binary predictions
# Print the classification report
print(classification_report(y_test, test_pred))
# Create the confusion matrix
cm = confusion_matrix(y_test, test_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/63 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
63/63 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step precision recall f1-score support 0.0 0.94 0.60 0.73 1593 1.0 0.35 0.86 0.50 407 accuracy 0.65 2000 macro avg 0.65 0.73 0.61 2000 weighted avg 0.82 0.65 0.68 2000
a larger portion of customers predicted as churners are actually not going to churn
# Generate predictions on the test set
test_pred = model_23.predict(X_test)
test_pred = (test_pred > 0.4) # Convert probabilities to binary predictions
# Print the classification report
print(classification_report(y_test, test_pred))
# Create the confusion matrix
cm = confusion_matrix(y_test, test_pred)
# Plot the confusion matrix as a heatmap
plt.figure(figsize=(8, 5))
sns.heatmap(cm, annot=True, fmt='.0f',
xticklabels=['Not Churned', 'Churned'],
yticklabels=['Not Churned', 'Churned'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
11/63 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step
/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn( /usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/structured_function.py:258: UserWarning: Even though the `tf.config.experimental_run_functions_eagerly` option is set, this option does not apply to tf.data functions. To force eager execution of tf.data functions, please use `tf.data.experimental.enable_debug_mode()`. warnings.warn(
63/63 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step precision recall f1-score support 0.0 0.93 0.72 0.81 1593 1.0 0.42 0.79 0.55 407 accuracy 0.74 2000 macro avg 0.68 0.76 0.68 2000 weighted avg 0.83 0.74 0.76 2000
we should stick with the .5 threshold... otherwise a lot of our errors move over too the very dangerous predicted churn actually not churn category
Actionable Insights and Business Recommendations¶
we need do develop retenntion starategies:
Offer incentives such as providing discounts or loyalty programs of different kinds.
ideally these offers should be personalized or segmented/clustered
Improve Customer Experience: Address potential pain points identified through the data analysis.
Proactive Communication: Reach out to these customers before they churn to understand their concerns and offer solutions.
we also need to do more Feature Engineering to improve model accuracy and reduce false positives.
collect more data
keep an eye on model performance
Power Ahead